From ec82ba398eb657fd2d01cf8a499d3b28a59a9254 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 23 Mar 2026 13:15:16 +0100 Subject: [PATCH 01/28] Document the 'repro' tag in README --- README.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 928a6b9..132bb40 100644 --- a/README.md +++ b/README.md @@ -12,11 +12,16 @@ Arch Linux provides OCI-Compliant container images in multiple repositories: * [Daily in our ghcr.io repository][ghcr-containers]: `podman pull ghcr.io/archlinux/archlinux:latest` or `docker pull ghcr.io/archlinux/archlinux:latest` -Three versions of the image are provided: `base` (approx. 150 MiB), `base-devel` +Four versions of the image are provided: `base` (approx. 150 MiB), `base-devel` (approx. 260 MiB) and `multilib-devel` (approx. 300MiB) containing the -respective meta package. All of them are available as -tags with `latest` pointing to `base`. Additionally, images are tagged with their -date and build job number, f.e. `base-devel-20201118.0.9436`. +respective meta package; and `repro` which is a bit for bit reproducible image +based on the `base` version (note that, to ensure reproducibility, the pacman keys +are stripped from this image so you're expected to run +`pacman-key --init && pacman-key --populate archlinux` before being able to update +the system and install packages via `pacman`). +All of them are available as tags with `latest` pointing to `base`. +Additionally, images are tagged with their date and build job number, +f.e. `base-devel-20201118.0.9436`. While the images are regularly kept up to date it is strongly recommended running `pacman -Syu` right after starting a container due to the rolling @@ -35,7 +40,7 @@ $ cosign verify ghcr.io/archlinux/archlinux:latest --certificate-identity-regexp * Provide the Arch experience in a Docker image * Provide the simplest but complete image to `base`, `base-devel` and `multilib-devel` on a regular basis -* `pacman` needs to work out of the box +* `pacman` needs to work out of the box (with the expection of the `repro` image for now, due to technical constraints) * All installed packages have to be kept unmodified >>> @@ -67,7 +72,7 @@ Make sure your user can directly interact with Podman (i.e. `podman info` works) ### Usage There are multiple `make image-XXX` targets, where each creates the respective `archlinux:XXX` image based on the corresponding meta package. -Currently those include `base`, `base-devel` and `multilib-devel`. +Currently those include `base`, `base-devel`, `multilib-devel` and `repro`. ## Pipeline @@ -118,4 +123,4 @@ Every year in June the content of the protected `GITLAB_PROJECT_TOKEN` variable [self-container-registry]: https://gitlab.archlinux.org/archlinux/archlinux-docker/container_registry -[access-tokens]: https://gitlab.archlinux.org/archlinux/archlinux-docker/-/settings/access_tokens \ No newline at end of file +[access-tokens]: https://gitlab.archlinux.org/archlinux/archlinux-docker/-/settings/access_tokens From a4bf94ea083cac51207e97ff536fda5438ba3dad Mon Sep 17 00:00:00 2001 From: Mark Hegreberg Date: Mon, 23 Mar 2026 09:01:03 -0700 Subject: [PATCH 02/28] fix typo and slight wording change --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 132bb40..72bc0dc 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ $ cosign verify ghcr.io/archlinux/archlinux:latest --certificate-identity-regexp * Provide the Arch experience in a Docker image * Provide the simplest but complete image to `base`, `base-devel` and `multilib-devel` on a regular basis -* `pacman` needs to work out of the box (with the expection of the `repro` image for now, due to technical constraints) +* `pacman` needs to work out of the box (with the exception of the `repro` image, while we are working on technical contstrains) * All installed packages have to be kept unmodified >>> From 20bbc94b91cc50f99b54a3891e6d869d4e8f2f96 Mon Sep 17 00:00:00 2001 From: Mark Hegreberg Date: Fri, 27 Mar 2026 19:27:56 -0700 Subject: [PATCH 03/28] repro POC this commit takes the relevant repro steps from the wsl image, and wraps breaking changes to only affect the :repro image testing reproducability is not yet included, so we can discuss the approach first --- .gitlab-ci.yml | 25 +++++++++++++++++++------ Makefile | 6 ++++-- scripts/make-rootfs.sh | 23 ++++++++++++++++++++++- 3 files changed, 45 insertions(+), 9 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 20e75e7..bf0eb0b 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -61,7 +61,7 @@ get_version: stage: rootfs parallel: matrix: - - GROUP: [base, base-devel, multilib-devel] + - GROUP: [repro, base, base-devel, multilib-devel] before_script: - pacman -Syu --noconfirm git make fakechroot fakeroot - pacman -Sdd --noconfirm devtools @@ -98,7 +98,7 @@ rootfs:secure: stage: image parallel: matrix: - - GROUP: [base, base-devel, multilib-devel] + - GROUP: [repro, base, base-devel, multilib-devel] tags: - vm id_tokens: @@ -192,6 +192,12 @@ image:publish:secure: - id -u http - locale | grep -q UTF-8 +test:repro: + extends: .test + image: $CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG + script: + - *test-script + test:base: extends: .test image: $CI_REGISTRY_IMAGE:base-$CI_COMMIT_REF_SLUG @@ -234,7 +240,7 @@ pre-release: -d "{\"full_description\": $(cat README.md | jq -sR .)}" # Upload rootfs to the Generic Packages Repository - for group in base base-devel multilib-devel; do + for group in repro base base-devel multilib-devel; do rootfs_file="${group}-${BUILD_VERSION}.tar.zst" mv "output/${group}.tar.zst" "output/${rootfs_file}" mv "output/${group}.tar.zst.SHA256" "output/${rootfs_file}.SHA256" @@ -246,7 +252,7 @@ pre-release: done # Create the Dockerfiles, commit to the release branch - for group in base base-devel multilib-devel; do + for group in repro base base-devel multilib-devel; do rootfs_file="${group}-${BUILD_VERSION}.tar.zst" ./scripts/make-dockerfile.sh "${rootfs_file}" "${group}" "output" "curl -sOJL \"${PACKAGE_REGISTRY_URL}/${rootfs_file}\"" "${group}" sed -i "/^COPY ${rootfs_file} \/$/d" output/Dockerfile.${group} @@ -258,6 +264,9 @@ pre-release: --form "branch=releases" --form "commit_message=Release ${BUILD_VERSION}" --form "actions[][action]=update" + --form "actions[][file_path]=Dockerfile.repro" + --form "actions[][content]=> library/archlinux echo "GitCommit: ${BUILD_COMMIT}" >> library/archlinux diff --git a/Makefile b/Makefile index 86d9c72..7a6361d 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,8 @@ OCITOOL=podman # or docker BUILDDIR=$(shell pwd)/build OUTPUTDIR=$(shell pwd)/output +ARCHIVE_SNAPSHOT=$(shell date -d "$(awk -F. '{print $1"-"$2"-"$3}' <<< "$IMAGE_VERSION") -1 day" +"%Y/%m/%d") +SOURCE_DATE_EPOCH=$(shell date -u -d "$(echo "$ARCHIVE_SNAPSHOT")" +"%s") .PHONY: clean clean: @@ -8,7 +10,7 @@ clean: .PRECIOUS: $(OUTPUTDIR)/%.tar.zst $(OUTPUTDIR)/%.tar.zst: - scripts/make-rootfs.sh $(*) $(BUILDDIR) $(OUTPUTDIR) + scripts/make-rootfs.sh $(*) $(BUILDDIR) $(OUTPUTDIR) $(ARCHIVE_SNAPSHOT) $(SOURCE_DATE_EPOCH) .PRECIOUS: $(OUTPUTDIR)/Dockerfile.% $(OUTPUTDIR)/Dockerfile.%: $(OUTPUTDIR)/%.tar.zst @@ -16,6 +18,6 @@ $(OUTPUTDIR)/Dockerfile.%: $(OUTPUTDIR)/%.tar.zst # The following is for local builds only, it is not used by the CI/CD pipeline -all: image-base image-base-devel image-multilib-devel +all: image-repro image-base image-base-devel image-multilib-devel image-%: $(OUTPUTDIR)/Dockerfile.% ${OCITOOL} build -f $(OUTPUTDIR)/Dockerfile.$(*) -t archlinux/archlinux:$(*) $(OUTPUTDIR) diff --git a/scripts/make-rootfs.sh b/scripts/make-rootfs.sh index a55f718..fc9c996 100755 --- a/scripts/make-rootfs.sh +++ b/scripts/make-rootfs.sh @@ -7,6 +7,8 @@ declare -r WRAPPER="fakechroot -- fakeroot" declare -r GROUP="$1" declare -r BUILDDIR="$2" declare -r OUTPUTDIR="$3" +declare -r ARCHIVE_SNAPSHOT="$4" +declare -rx SOURCE_DATE_EPOCH="$5" mkdir -vp "$BUILDDIR/alpm-hooks/usr/share/libalpm/hooks" find /usr/share/libalpm/hooks -exec ln -sf /dev/null "$BUILDDIR/alpm-hooks"{} \; @@ -33,18 +35,33 @@ fi cp --recursive --preserve=timestamps rootfs/* "$BUILDDIR/" ln -fs /usr/lib/os-release "$BUILDDIR/etc/os-release" +# Use archived repo snapshot from archive.archlinux.org for reproducible builds +sed -i "1iServer = https://archive.archlinux.org/repos/$ARCHIVE_SNAPSHOT/\\\$repo/os/\\\$arch" "$BUILDDIR/etc/pacman.d/mirrorlist" + $WRAPPER -- \ pacman -Sy -r "$BUILDDIR" \ --disable-sandbox-filesystem \ --noconfirm --dbpath "$BUILDDIR/var/lib/pacman" \ --config pacman.conf \ --noscriptlet \ - --hookdir "$BUILDDIR/alpm-hooks/usr/share/libalpm/hooks/" base "$GROUP" + --hookdir "$BUILDDIR/alpm-hooks/usr/share/libalpm/hooks/" base ${GROUP:+${GROUP/repro/}} +# # repro is not a package, so excluded here ^ $WRAPPER -- chroot "$BUILDDIR" update-ca-trust $WRAPPER -- chroot "$BUILDDIR" pacman-key --init $WRAPPER -- chroot "$BUILDDIR" pacman-key --populate +# Remove archived repo snapshot from the mirrorlist +sed -i '1d' "$BUILDDIR/etc/pacman.d/mirrorlist" + +if [[ "$GROUP" == "repro" ]]; + # Clear pacman keyring for reproducible builds + rm -rf "$BUILDDIR"/etc/pacman.d/gnupg/* +fi + +# Normalize mtimes +find "$BUILDDIR" -exec touch --no-dereference --date="@$SOURCE_DATE_EPOCH" {} + + # add system users $WRAPPER -- chroot "$BUILDDIR" /usr/bin/systemd-sysusers --root "/" @@ -58,6 +75,10 @@ fakeroot -- \ --numeric-owner \ --xattrs \ --acls \ + --mtime="@$SOURCE_DATE_EPOCH" \ + --clamp-mtime \ + --sort=name \ + --pax-option=delete=atime,delete=ctime \ --exclude-from=exclude \ -C "$BUILDDIR" \ -c . \ From 7fe6027fa967955546c1b79348cb9fc507178dbc Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Sat, 28 Mar 2026 14:35:55 +0100 Subject: [PATCH 04/28] Move repro specific steps under the repro group condition Given that we intend to create a dedicated repro tag, we should probably put every repro steps behind this condition and leave the other groups / tags untouched for now. --- scripts/make-rootfs.sh | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/scripts/make-rootfs.sh b/scripts/make-rootfs.sh index fc9c996..0f0cf9b 100755 --- a/scripts/make-rootfs.sh +++ b/scripts/make-rootfs.sh @@ -36,7 +36,9 @@ cp --recursive --preserve=timestamps rootfs/* "$BUILDDIR/" ln -fs /usr/lib/os-release "$BUILDDIR/etc/os-release" # Use archived repo snapshot from archive.archlinux.org for reproducible builds -sed -i "1iServer = https://archive.archlinux.org/repos/$ARCHIVE_SNAPSHOT/\\\$repo/os/\\\$arch" "$BUILDDIR/etc/pacman.d/mirrorlist" +if [[ "$GROUP" == "repro" ]]; then + sed -i "1iServer = https://archive.archlinux.org/repos/$ARCHIVE_SNAPSHOT/\\\$repo/os/\\\$arch" "$BUILDDIR/etc/pacman.d/mirrorlist" +fi $WRAPPER -- \ pacman -Sy -r "$BUILDDIR" \ @@ -51,23 +53,30 @@ $WRAPPER -- chroot "$BUILDDIR" update-ca-trust $WRAPPER -- chroot "$BUILDDIR" pacman-key --init $WRAPPER -- chroot "$BUILDDIR" pacman-key --populate -# Remove archived repo snapshot from the mirrorlist -sed -i '1d' "$BUILDDIR/etc/pacman.d/mirrorlist" - -if [[ "$GROUP" == "repro" ]]; +if [[ "$GROUP" == "repro" ]]; then + # Remove archived repo snapshot from the mirrorlist + sed -i '1d' "$BUILDDIR/etc/pacman.d/mirrorlist" # Clear pacman keyring for reproducible builds rm -rf "$BUILDDIR"/etc/pacman.d/gnupg/* + # Normalize mtimes + find "$BUILDDIR" -exec touch --no-dereference --date="@$SOURCE_DATE_EPOCH" {} + fi -# Normalize mtimes -find "$BUILDDIR" -exec touch --no-dereference --date="@$SOURCE_DATE_EPOCH" {} + - # add system users $WRAPPER -- chroot "$BUILDDIR" /usr/bin/systemd-sysusers --root "/" # remove passwordless login for root (see CVE-2019-5021 for reference) sed -i -e 's/^root::/root:!:/' "$BUILDDIR/etc/shadow" +if [[ "$GROUP" == "repro" ]]; then + repro_tar_options=( + --mtime="@$SOURCE_DATE_EPOCH" + --clamp-mtime + --sort=name + --pax-option=delete=atime,delete=ctime + ) +fi + # fakeroot to map the gid/uid of the builder process to root # fixes #22 fakeroot -- \ @@ -75,10 +84,7 @@ fakeroot -- \ --numeric-owner \ --xattrs \ --acls \ - --mtime="@$SOURCE_DATE_EPOCH" \ - --clamp-mtime \ - --sort=name \ - --pax-option=delete=atime,delete=ctime \ + "${repro_tar_options[@]}" \ --exclude-from=exclude \ -C "$BUILDDIR" \ -c . \ From b21717021cd5e6f77deb0200c52a6b3e03e00b88 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Sat, 28 Mar 2026 14:39:41 +0100 Subject: [PATCH 05/28] Update comment styling --- scripts/make-rootfs.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/scripts/make-rootfs.sh b/scripts/make-rootfs.sh index 0f0cf9b..bd3a1c1 100755 --- a/scripts/make-rootfs.sh +++ b/scripts/make-rootfs.sh @@ -46,8 +46,7 @@ $WRAPPER -- \ --noconfirm --dbpath "$BUILDDIR/var/lib/pacman" \ --config pacman.conf \ --noscriptlet \ - --hookdir "$BUILDDIR/alpm-hooks/usr/share/libalpm/hooks/" base ${GROUP:+${GROUP/repro/}} -# # repro is not a package, so excluded here ^ + --hookdir "$BUILDDIR/alpm-hooks/usr/share/libalpm/hooks/" base ${GROUP:+${GROUP/repro/}} # repro is not a package $WRAPPER -- chroot "$BUILDDIR" update-ca-trust $WRAPPER -- chroot "$BUILDDIR" pacman-key --init From 4f4495e15bf11b6fc0862928cea021680d2686bc Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Sat, 28 Mar 2026 14:47:16 +0100 Subject: [PATCH 06/28] Fix call to unexisting var in Makefile --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 7a6361d..09b1159 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ OCITOOL=podman # or docker BUILDDIR=$(shell pwd)/build OUTPUTDIR=$(shell pwd)/output -ARCHIVE_SNAPSHOT=$(shell date -d "$(awk -F. '{print $1"-"$2"-"$3}' <<< "$IMAGE_VERSION") -1 day" +"%Y/%m/%d") +ARCHIVE_SNAPSHOT=$(shell date -d "-1 day" +"%Y/%m/%d") SOURCE_DATE_EPOCH=$(shell date -u -d "$(echo "$ARCHIVE_SNAPSHOT")" +"%s") .PHONY: clean From c4462ed40b2fbf7b8932208742e2492d05b6ada9 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Sat, 28 Mar 2026 14:53:14 +0100 Subject: [PATCH 07/28] Fix ordering in GitLab CI and Makefile --- .gitlab-ci.yml | 36 ++++++++++++++++++------------------ Makefile | 2 +- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index bf0eb0b..42d5e7e 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -61,7 +61,7 @@ get_version: stage: rootfs parallel: matrix: - - GROUP: [repro, base, base-devel, multilib-devel] + - GROUP: [base, base-devel, multilib-devel, repro] before_script: - pacman -Syu --noconfirm git make fakechroot fakeroot - pacman -Sdd --noconfirm devtools @@ -98,7 +98,7 @@ rootfs:secure: stage: image parallel: matrix: - - GROUP: [repro, base, base-devel, multilib-devel] + - GROUP: [base, base-devel, multilib-devel, repro] tags: - vm id_tokens: @@ -192,12 +192,6 @@ image:publish:secure: - id -u http - locale | grep -q UTF-8 -test:repro: - extends: .test - image: $CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG - script: - - *test-script - test:base: extends: .test image: $CI_REGISTRY_IMAGE:base-$CI_COMMIT_REF_SLUG @@ -215,6 +209,12 @@ test:base-devel: - test -u /usr/bin/sudo # issue 70 - test -u /usr/bin/passwd +test:repro: + extends: .test + image: $CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG + script: + - *test-script + pre-release: stage: pre-release image: registry.gitlab.com/gitlab-org/release-cli:latest @@ -240,7 +240,7 @@ pre-release: -d "{\"full_description\": $(cat README.md | jq -sR .)}" # Upload rootfs to the Generic Packages Repository - for group in repro base base-devel multilib-devel; do + for group in base base-devel multilib-devel repro; do rootfs_file="${group}-${BUILD_VERSION}.tar.zst" mv "output/${group}.tar.zst" "output/${rootfs_file}" mv "output/${group}.tar.zst.SHA256" "output/${rootfs_file}.SHA256" @@ -252,7 +252,7 @@ pre-release: done # Create the Dockerfiles, commit to the release branch - for group in repro base base-devel multilib-devel; do + for group in base base-devel multilib-devel repro; do rootfs_file="${group}-${BUILD_VERSION}.tar.zst" ./scripts/make-dockerfile.sh "${rootfs_file}" "${group}" "output" "curl -sOJL \"${PACKAGE_REGISTRY_URL}/${rootfs_file}\"" "${group}" sed -i "/^COPY ${rootfs_file} \/$/d" output/Dockerfile.${group} @@ -264,9 +264,6 @@ pre-release: --form "branch=releases" --form "commit_message=Release ${BUILD_VERSION}" --form "actions[][action]=update" - --form "actions[][file_path]=Dockerfile.repro" - --form "actions[][content]=> library/archlinux echo "GitCommit: ${BUILD_COMMIT}" >> library/archlinux diff --git a/Makefile b/Makefile index 09b1159..482707d 100644 --- a/Makefile +++ b/Makefile @@ -18,6 +18,6 @@ $(OUTPUTDIR)/Dockerfile.%: $(OUTPUTDIR)/%.tar.zst # The following is for local builds only, it is not used by the CI/CD pipeline -all: image-repro image-base image-base-devel image-multilib-devel +all: image-base image-base-devel image-multilib-devel image-repro image-%: $(OUTPUTDIR)/Dockerfile.% ${OCITOOL} build -f $(OUTPUTDIR)/Dockerfile.$(*) -t archlinux/archlinux:$(*) $(OUTPUTDIR) From 2c15b530fe02f819d9301c0f1a4794737f6eae91 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Sat, 28 Mar 2026 14:55:41 +0100 Subject: [PATCH 08/28] Syntax fix --- .gitlab-ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 42d5e7e..2bb41ab 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -310,7 +310,7 @@ release: ref: 'releases' assets: links: - - name: 'base-${BUILD_VERSION}.tar.zst + - name: 'base-${BUILD_VERSION}.tar.zst' url: '${PACKAGE_REGISTRY_URL}/base-${BUILD_VERSION}.tar.zst' - name: 'base-${BUILD_VERSION}.tar.zst.SHA256' url: '${PACKAGE_REGISTRY_URL}/base-${BUILD_VERSION}.tar.zst.SHA256' @@ -322,7 +322,7 @@ release: url: '${PACKAGE_REGISTRY_URL}/multilib-devel-${BUILD_VERSION}.tar.zst' - name: 'multilib-devel-${BUILD_VERSION}.tar.zst.SHA256' url: '${PACKAGE_REGISTRY_URL}/multilib-devel-${BUILD_VERSION}.tar.zst.SHA256' - - name: 'repro-${BUILD_VERSION}.tar.zst + - name: 'repro-${BUILD_VERSION}.tar.zst' url: '${PACKAGE_REGISTRY_URL}/repro-${BUILD_VERSION}.tar.zst' - name: 'repro-${BUILD_VERSION}.tar.zst.SHA256' url: '${PACKAGE_REGISTRY_URL}/repro-${BUILD_VERSION}.tar.zst.SHA256' From 2f44c1aeba6f395057c5ce4329fb8b91a8edf8d0 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Sat, 28 Mar 2026 15:19:17 +0100 Subject: [PATCH 09/28] Re-generate pacman keys before testing the repro image in CI --- .gitlab-ci.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 2bb41ab..12c32b5 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -213,6 +213,8 @@ test:repro: extends: .test image: $CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG script: + - pacman-key --init + - pacman-key --populate archlinux - *test-script pre-release: From 0e2fd8ee15abc460af89ed664c15efef9b5db01a Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Sat, 28 Mar 2026 15:24:24 +0100 Subject: [PATCH 10/28] Run arch-repro-status when testing the 'repro' image This is more informative than anything, we're primarily looking at providing a bit for bit reproducible image. The reproducibility of the userspace is not fully guaranteed at the moment --- .gitlab-ci.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 12c32b5..a1f7a4a 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -215,6 +215,8 @@ test:repro: script: - pacman-key --init - pacman-key --populate archlinux + - pacman -Syu --noconfirm arch-repro-status + - arch-repro-status - *test-script pre-release: From cd87d3eb612d5bc8a660a19e7a7bb9e1bf5c252b Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 30 Mar 2026 13:14:59 +0200 Subject: [PATCH 11/28] WIP --- .gitlab-ci.yml | 6 ++++++ Makefile | 9 ++++++++- scripts/make-repro-image.sh | 17 +++++++++++++++++ 3 files changed, 31 insertions(+), 1 deletion(-) create mode 100755 scripts/make-repro-image.sh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index a1f7a4a..ca46fce 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -122,6 +122,9 @@ image:build: - pacman -Syu --noconfirm podman - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" - 'echo -e "default-docker:\n use-sigstore-attachments: true" > /etc/containers/registries.d/sigstore.yaml' + artifacts: + paths: + - output/* image:build:secure: extends: .image @@ -137,6 +140,9 @@ image:build:secure: - pacman -Syu --noconfirm podman - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" - 'echo -e "default-docker:\n use-sigstore-attachments: true" > /etc/containers/registries.d/sigstore.yaml' + artifacts: + paths: + - output/* # Build and publish to the Arch Linux group namespaces: # https://hub.docker.com/r/archlinux/archlinux diff --git a/Makefile b/Makefile index 482707d..7304954 100644 --- a/Makefile +++ b/Makefile @@ -1,12 +1,14 @@ OCITOOL=podman # or docker BUILDDIR=$(shell pwd)/build +REPRO_BUILDDIR=$(shell pwd)/repro-build OUTPUTDIR=$(shell pwd)/output +REPRO_OUTPUTDIR=$(shell pwd)/repro-output ARCHIVE_SNAPSHOT=$(shell date -d "-1 day" +"%Y/%m/%d") SOURCE_DATE_EPOCH=$(shell date -u -d "$(echo "$ARCHIVE_SNAPSHOT")" +"%s") .PHONY: clean clean: - rm -rf $(BUILDDIR) $(OUTPUTDIR) + rm -rf $(BUILDDIR) $(REPRO_BUILDDIR) $(OUTPUTDIR) $(REPRO_OUTPUTDIR) .PRECIOUS: $(OUTPUTDIR)/%.tar.zst $(OUTPUTDIR)/%.tar.zst: @@ -16,6 +18,11 @@ $(OUTPUTDIR)/%.tar.zst: $(OUTPUTDIR)/Dockerfile.%: $(OUTPUTDIR)/%.tar.zst scripts/make-dockerfile.sh "$(*).tar.zst" $(*) $(OUTPUTDIR) "true" "Dev" +# The following aims to rebuild a "repro" tagged image and verify the reproducibility status + +repro: + scripts/make-repro.sh $(*) $(OUTPUTDIR) $(REPRO_BUILDDIR) $(REPRO_OUTPUTDIR) $(ARCHIVE_SNAPSHOT) $(SOURCE_DATE_EPOCH) + # The following is for local builds only, it is not used by the CI/CD pipeline all: image-base image-base-devel image-multilib-devel image-repro diff --git a/scripts/make-repro-image.sh b/scripts/make-repro-image.sh new file mode 100755 index 0000000..6a07da6 --- /dev/null +++ b/scripts/make-repro-image.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +set -euo pipefail + +declare -r ORIG_OUTPUTDIR="$1" +declare -r REPRO_BUILDDIR="$2" +declare -r REPRO_OUTPUTDIR="$3" +declare -r ARCHIVE_SNAPSHOT="$4" +declare -rx SOURCE_DATE_EPOCH="$5" + +echo -e "\n-- Testing the image reproducibility --\n" +make BUILDDIR="$REPRO_BUILDDIR" OUTPUTDIR="$REPRO_OUTPUTDIR" ARCHIVE_SNAPSHOT="$ARCHIVE_SNAPSHOT" SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" +echo "The sha256 hash of the original image is:" +sha256sums "$ORIG_OUTPUTDIR/" +echo "The sha256 hash of the reproduced image is:" +sha256sums "$REPRO_OUTPUTDIR/" +diffoscope "$ORIG_OUTPUTDIR/" "$REPRO_OUTPUTDIR/" && echo -e "\nImage is reproducible!" From 7069a6cc06bcff2ebd1805e4a64f8041d8b61dfb Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 30 Mar 2026 20:26:23 +0200 Subject: [PATCH 12/28] Revert "WIP" This reverts commit cd87d3eb612d5bc8a660a19e7a7bb9e1bf5c252b. --- .gitlab-ci.yml | 6 ------ Makefile | 9 +-------- scripts/make-repro-image.sh | 17 ----------------- 3 files changed, 1 insertion(+), 31 deletions(-) delete mode 100755 scripts/make-repro-image.sh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index ca46fce..a1f7a4a 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -122,9 +122,6 @@ image:build: - pacman -Syu --noconfirm podman - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" - 'echo -e "default-docker:\n use-sigstore-attachments: true" > /etc/containers/registries.d/sigstore.yaml' - artifacts: - paths: - - output/* image:build:secure: extends: .image @@ -140,9 +137,6 @@ image:build:secure: - pacman -Syu --noconfirm podman - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" - 'echo -e "default-docker:\n use-sigstore-attachments: true" > /etc/containers/registries.d/sigstore.yaml' - artifacts: - paths: - - output/* # Build and publish to the Arch Linux group namespaces: # https://hub.docker.com/r/archlinux/archlinux diff --git a/Makefile b/Makefile index 7304954..482707d 100644 --- a/Makefile +++ b/Makefile @@ -1,14 +1,12 @@ OCITOOL=podman # or docker BUILDDIR=$(shell pwd)/build -REPRO_BUILDDIR=$(shell pwd)/repro-build OUTPUTDIR=$(shell pwd)/output -REPRO_OUTPUTDIR=$(shell pwd)/repro-output ARCHIVE_SNAPSHOT=$(shell date -d "-1 day" +"%Y/%m/%d") SOURCE_DATE_EPOCH=$(shell date -u -d "$(echo "$ARCHIVE_SNAPSHOT")" +"%s") .PHONY: clean clean: - rm -rf $(BUILDDIR) $(REPRO_BUILDDIR) $(OUTPUTDIR) $(REPRO_OUTPUTDIR) + rm -rf $(BUILDDIR) $(OUTPUTDIR) .PRECIOUS: $(OUTPUTDIR)/%.tar.zst $(OUTPUTDIR)/%.tar.zst: @@ -18,11 +16,6 @@ $(OUTPUTDIR)/%.tar.zst: $(OUTPUTDIR)/Dockerfile.%: $(OUTPUTDIR)/%.tar.zst scripts/make-dockerfile.sh "$(*).tar.zst" $(*) $(OUTPUTDIR) "true" "Dev" -# The following aims to rebuild a "repro" tagged image and verify the reproducibility status - -repro: - scripts/make-repro.sh $(*) $(OUTPUTDIR) $(REPRO_BUILDDIR) $(REPRO_OUTPUTDIR) $(ARCHIVE_SNAPSHOT) $(SOURCE_DATE_EPOCH) - # The following is for local builds only, it is not used by the CI/CD pipeline all: image-base image-base-devel image-multilib-devel image-repro diff --git a/scripts/make-repro-image.sh b/scripts/make-repro-image.sh deleted file mode 100755 index 6a07da6..0000000 --- a/scripts/make-repro-image.sh +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -declare -r ORIG_OUTPUTDIR="$1" -declare -r REPRO_BUILDDIR="$2" -declare -r REPRO_OUTPUTDIR="$3" -declare -r ARCHIVE_SNAPSHOT="$4" -declare -rx SOURCE_DATE_EPOCH="$5" - -echo -e "\n-- Testing the image reproducibility --\n" -make BUILDDIR="$REPRO_BUILDDIR" OUTPUTDIR="$REPRO_OUTPUTDIR" ARCHIVE_SNAPSHOT="$ARCHIVE_SNAPSHOT" SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" -echo "The sha256 hash of the original image is:" -sha256sums "$ORIG_OUTPUTDIR/" -echo "The sha256 hash of the reproduced image is:" -sha256sums "$REPRO_OUTPUTDIR/" -diffoscope "$ORIG_OUTPUTDIR/" "$REPRO_OUTPUTDIR/" && echo -e "\nImage is reproducible!" From 6103dcbc5fb777afb35e5bed9943e64733cd3b11 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 30 Mar 2026 21:00:36 +0200 Subject: [PATCH 13/28] Add repro-test stage to the CI Rebuild the rootFS and the "repro" image, pull the originally built "repro" image and compare them (with `podman digest` and `diffoci`). --- .gitlab-ci.yml | 63 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index a1f7a4a..522fe66 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -6,6 +6,7 @@ stages: - lint - rootfs - image + - repro - test - pre-release - release @@ -105,9 +106,16 @@ rootfs:secure: SIGSTORE_ID_TOKEN: aud: sigstore script: - - podman build - -f "$CI_PROJECT_DIR/output/Dockerfile.$GROUP" - -t "$CI_REGISTRY_IMAGE:$GROUP-$CI_COMMIT_REF_SLUG" + - | + REPRO_ARGS="" + if [ "$GROUP" = "repro" ]; then + SOURCE_DATE_EPOCH=$(date -u -d "-1 day" +%s) + REPRO_ARGS="--source-date-epoch=${SOURCE_DATE_EPOCH} --rewrite-timestamp" + fi + podman build \ + $REPRO_ARGS \ + -f "$CI_PROJECT_DIR/output/Dockerfile.$GROUP" \ + -t "$CI_REGISTRY_IMAGE:$GROUP-$CI_COMMIT_REF_SLUG" \ "$CI_PROJECT_DIR/output" - podman push --sign-by-sigstore=<(sed "s/TEMPLATE_OIDC_ID_TOKEN/${SIGSTORE_ID_TOKEN}/" sigstore-param-file.yaml) "$CI_REGISTRY_IMAGE:$GROUP-$CI_COMMIT_REF_SLUG" @@ -138,6 +146,55 @@ image:build:secure: - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" - 'echo -e "default-docker:\n use-sigstore-attachments: true" > /etc/containers/registries.d/sigstore.yaml' +.test_repro: + stage: repro + before_script: + - pacman -Syu --noconfirm git make fakechroot fakeroot podman diffoci + - pacman -Sdd --noconfirm devtools + script: + - make BUILDDIR="$PWD/repro-build" OUTPUTDIR="$PWD/repro-output" $PWD/repro-output/Dockerfile.repro + - | + SOURCE_DATE_EPOCH=$(date -u -d "-1 day" +%s) + podman build \ + --no-cache \ + --source-date-epoch=${SOURCE_DATE_EPOCH} \ + --rewrite-timestamp \ + -f "$CI_PROJECT_DIR/repro-output/Dockerfile.repro" \ + -t "archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG" \ + "$CI_PROJECT_DIR/repro-output" + - podman pull "$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG" + - echo "Digest of the original image is:" + - podman inspect --format '{{.Digest}}' "$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG" + - echo "Digest of the rebuilt image is:" + - podman inspect --format '{{.Digest}}' "localhost/archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG" + - diffoci diff --semantic podman://$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG podman://localhost/archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG + - echo "Image is reproducible!" + artifacts: + paths: + - repro-output/* + exclude: + - repro-output/*.tar + expire_in: 2h + +test-repro: + extends: .test_repro + tags: + - vm + except: + - master@archlinux/archlinux-docker + - releases@archlinux/archlinux-docker + - schedules@archlinux/archlinux-docker + - tags@archlinux/archlinux-docker + +test-repro:secure: + extends: .test_repro + tags: + - secure + - vm + only: + - master@archlinux/archlinux-docker + - schedules@archlinux/archlinux-docker + # Build and publish to the Arch Linux group namespaces: # https://hub.docker.com/r/archlinux/archlinux # https://quay.io/repository/archlinux/archlinux From 87a723680eeef81c5c9bbe74af066a4f337b72ba Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 30 Mar 2026 22:39:15 +0200 Subject: [PATCH 14/28] Get rid of pacman logs for repro image --- scripts/make-rootfs.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/make-rootfs.sh b/scripts/make-rootfs.sh index bd3a1c1..6359de5 100755 --- a/scripts/make-rootfs.sh +++ b/scripts/make-rootfs.sh @@ -38,6 +38,9 @@ ln -fs /usr/lib/os-release "$BUILDDIR/etc/os-release" # Use archived repo snapshot from archive.archlinux.org for reproducible builds if [[ "$GROUP" == "repro" ]]; then sed -i "1iServer = https://archive.archlinux.org/repos/$ARCHIVE_SNAPSHOT/\\\$repo/os/\\\$arch" "$BUILDDIR/etc/pacman.d/mirrorlist" + repro_pacman_options=( + --logfile /dev/null + ) fi $WRAPPER -- \ @@ -46,6 +49,7 @@ $WRAPPER -- \ --noconfirm --dbpath "$BUILDDIR/var/lib/pacman" \ --config pacman.conf \ --noscriptlet \ + "${repro_pacman_options[@]}" \ --hookdir "$BUILDDIR/alpm-hooks/usr/share/libalpm/hooks/" base ${GROUP:+${GROUP/repro/}} # repro is not a package $WRAPPER -- chroot "$BUILDDIR" update-ca-trust From eb80a94f54a83d4547e322251c69bfb575dc89e3 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 30 Mar 2026 22:47:09 +0200 Subject: [PATCH 15/28] Remove ldconfig cache from Dockerfile.template Not needed at runtime and adds non-determinism --- Dockerfile.template | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Dockerfile.template b/Dockerfile.template index 6ecfc3d..f7d3cf0 100644 --- a/Dockerfile.template +++ b/Dockerfile.template @@ -37,7 +37,8 @@ LABEL org.opencontainers.image.created="TEMPLATE_CREATED" COPY --from=verify /rootfs/ / RUN ldconfig && \ - sed -i '/BUILD_ID/a VERSION_ID=TEMPLATE_VERSION_ID' /etc/os-release + sed -i '/BUILD_ID/a VERSION_ID=TEMPLATE_VERSION_ID' /etc/os-release && \ + rm -f /var/cache/ldconfig/aux-cache ENV LANG=C.UTF-8 CMD ["/usr/bin/bash"] From 9a4c205f52d91d1411f297fd01d3476b95b51974 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 30 Mar 2026 23:01:08 +0200 Subject: [PATCH 16/28] Honor SDE in Dockerfile for the repro group --- Makefile | 2 +- scripts/make-dockerfile.sh | 10 +++++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 482707d..fabc38d 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,7 @@ $(OUTPUTDIR)/%.tar.zst: .PRECIOUS: $(OUTPUTDIR)/Dockerfile.% $(OUTPUTDIR)/Dockerfile.%: $(OUTPUTDIR)/%.tar.zst - scripts/make-dockerfile.sh "$(*).tar.zst" $(*) $(OUTPUTDIR) "true" "Dev" + scripts/make-dockerfile.sh "$(*).tar.zst" $(*) $(OUTPUTDIR) "true" "Dev" $(SOURCE_DATE_EPOCH) # The following is for local builds only, it is not used by the CI/CD pipeline diff --git a/scripts/make-dockerfile.sh b/scripts/make-dockerfile.sh index 878d18d..3404efc 100755 --- a/scripts/make-dockerfile.sh +++ b/scripts/make-dockerfile.sh @@ -7,16 +7,24 @@ declare -r GROUP="$2" declare -r OUTPUTDIR="$3" declare -r DOWNLOAD="$4" declare -r TITLE="$5" +declare -rx SOURCE_DATE_EPOCH="$6" # Do not use these directly in the sed below - it will mask git failures BUILD_VERSION="${BUILD_VERSION:-dev}" CI_COMMIT_SHA="${CI_COMMIT_SHA:-$(git rev-parse HEAD)}" +# Honor SOURCE_DATE_EPOCH for the repro GROUP +if [ "$GROUP" = "repro" ]; then + CREATED_TIMESTAMP=$(date -u -d "@$SOURCE_DATE_EPOCH" +%Y-%m-%dT%H:%M:%SZ) +else + CREATED_TIMESTAMP=$(date -Is) +fi + sed -e "s|TEMPLATE_ROOTFS_FILE|$ROOTFS_FILE|" \ -e "s|TEMPLATE_ROOTFS_DOWNLOAD|$DOWNLOAD|" \ -e "s|TEMPLATE_ROOTFS_HASH|$(cat $OUTPUTDIR/$ROOTFS_FILE.SHA256)|" \ -e "s|TEMPLATE_TITLE|Arch Linux $TITLE Image|" \ -e "s|TEMPLATE_VERSION_ID|$BUILD_VERSION|" \ -e "s|TEMPLATE_REVISION|$CI_COMMIT_SHA|" \ - -e "s|TEMPLATE_CREATED|$(date -Is)|" \ + -e "s|TEMPLATE_CREATED|$CREATED_TIMESTAMP|" \ Dockerfile.template > "$OUTPUTDIR/Dockerfile.$GROUP" From 8cefb71233e2e12fb75161934e03e42508b3181a Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Mon, 30 Mar 2026 23:23:14 +0200 Subject: [PATCH 17/28] Remove non-relevant digest checks The container registry is always going to rewrite parts of the image in an uncontrollable way. As far as we know, it's not possible to download a 1:1 copy of a build output from the container registry (until someone figures this out). As far as I understand it, it also explains why `diffoci --semantic` is a thing and why it's generally considered "good enough" (give current constraints). --- .gitlab-ci.yml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 522fe66..bd856a0 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -163,10 +163,6 @@ image:build:secure: -t "archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG" \ "$CI_PROJECT_DIR/repro-output" - podman pull "$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG" - - echo "Digest of the original image is:" - - podman inspect --format '{{.Digest}}' "$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG" - - echo "Digest of the rebuilt image is:" - - podman inspect --format '{{.Digest}}' "localhost/archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG" - diffoci diff --semantic podman://$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG podman://localhost/archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG - echo "Image is reproducible!" artifacts: From af4e991076be4f7bf35710e68baff0e8f6b4631d Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Tue, 31 Mar 2026 00:52:11 +0200 Subject: [PATCH 18/28] Consistency for if blocks style --- .gitlab-ci.yml | 2 +- scripts/make-dockerfile.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index bd856a0..4241153 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -108,7 +108,7 @@ rootfs:secure: script: - | REPRO_ARGS="" - if [ "$GROUP" = "repro" ]; then + if [[ "$GROUP" == "repro" ]]; then SOURCE_DATE_EPOCH=$(date -u -d "-1 day" +%s) REPRO_ARGS="--source-date-epoch=${SOURCE_DATE_EPOCH} --rewrite-timestamp" fi diff --git a/scripts/make-dockerfile.sh b/scripts/make-dockerfile.sh index 3404efc..aec8e97 100755 --- a/scripts/make-dockerfile.sh +++ b/scripts/make-dockerfile.sh @@ -14,7 +14,7 @@ BUILD_VERSION="${BUILD_VERSION:-dev}" CI_COMMIT_SHA="${CI_COMMIT_SHA:-$(git rev-parse HEAD)}" # Honor SOURCE_DATE_EPOCH for the repro GROUP -if [ "$GROUP" = "repro" ]; then +if [[ "$GROUP" == "repro" ]]; then CREATED_TIMESTAMP=$(date -u -d "@$SOURCE_DATE_EPOCH" +%Y-%m-%dT%H:%M:%SZ) else CREATED_TIMESTAMP=$(date -Is) From cccc73178e25e9f254b039555440235820183baa Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Tue, 31 Mar 2026 01:05:58 +0200 Subject: [PATCH 19/28] Add repro test for the rootFS Show bit for bit reproducibility of the rootFS --- .gitlab-ci.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4241153..be45a44 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -149,10 +149,16 @@ image:build:secure: .test_repro: stage: repro before_script: - - pacman -Syu --noconfirm git make fakechroot fakeroot podman diffoci + - pacman -Syu --noconfirm git make fakechroot fakeroot podman diffoscope diffoci - pacman -Sdd --noconfirm devtools script: - make BUILDDIR="$PWD/repro-build" OUTPUTDIR="$PWD/repro-output" $PWD/repro-output/Dockerfile.repro + - echo "The sha256 hash of the original rootFS is:" + - cat output/repro.tar.zst.SHA256 + - echo "The sha256 hash of the rebuilt rootFS is:" + - cat repro-output/repro.tar.zst.SHA256 + - diffoscope output/repro.tar.zst repro-output/repro.tar.zst + - echo "RootFS is reproducible!" - | SOURCE_DATE_EPOCH=$(date -u -d "-1 day" +%s) podman build \ From 65291543411e93c8eccedd4b5c3f19aa56facdb7 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Tue, 31 Mar 2026 09:51:35 +0200 Subject: [PATCH 20/28] Fix SDE definition for podman build and re-add digest comparison --- .gitlab-ci.yml | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index be45a44..09dfd3b 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -107,13 +107,15 @@ rootfs:secure: aud: sigstore script: - | - REPRO_ARGS="" if [[ "$GROUP" == "repro" ]]; then - SOURCE_DATE_EPOCH=$(date -u -d "-1 day" +%s) - REPRO_ARGS="--source-date-epoch=${SOURCE_DATE_EPOCH} --rewrite-timestamp" + SOURCE_DATE_EPOCH=$(date -u -d "today 00:00:00" +%s) + REPRO_ARGS=( + --source-date-epoch=${SOURCE_DATE_EPOCH} + --rewrite-timestamp + ) fi podman build \ - $REPRO_ARGS \ + "${REPRO_ARGS[@]}" \ -f "$CI_PROJECT_DIR/output/Dockerfile.$GROUP" \ -t "$CI_REGISTRY_IMAGE:$GROUP-$CI_COMMIT_REF_SLUG" \ "$CI_PROJECT_DIR/output" @@ -160,16 +162,20 @@ image:build:secure: - diffoscope output/repro.tar.zst repro-output/repro.tar.zst - echo "RootFS is reproducible!" - | - SOURCE_DATE_EPOCH=$(date -u -d "-1 day" +%s) + SOURCE_DATE_EPOCH=$(date -u -d "today 00:00:00" +%s) podman build \ --no-cache \ --source-date-epoch=${SOURCE_DATE_EPOCH} \ --rewrite-timestamp \ -f "$CI_PROJECT_DIR/repro-output/Dockerfile.repro" \ - -t "archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG" \ + -t "archlinux-docker:repro-$CI_COMMIT_REF_SLUG" \ "$CI_PROJECT_DIR/repro-output" - - podman pull "$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG" - - diffoci diff --semantic podman://$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG podman://localhost/archlinux:repro-rebuild-$CI_COMMIT_REF_SLUG + - podman pull $CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG + - echo "Digest of the original image is:" + - podman inspect --format '{{.Digest}}' "$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG" + - echo "Digest of the rebuilt image is:" + - podman inspect --format '{{.Digest}}' "localhost/archlinux-docker:repro-$CI_COMMIT_REF_SLUG" + - diffoci diff --semantic --verbose podman://$CI_REGISTRY_IMAGE:repro-$CI_COMMIT_REF_SLUG podman://localhost/archlinux-docker:repro-$CI_COMMIT_REF_SLUG - echo "Image is reproducible!" artifacts: paths: From 303235f6a547b031071136dec321f4229126a20e Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Wed, 1 Apr 2026 10:44:29 +0200 Subject: [PATCH 21/28] Use same SDE value for the rootFS and podman build --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index fabc38d..45c90a8 100644 --- a/Makefile +++ b/Makefile @@ -2,7 +2,7 @@ OCITOOL=podman # or docker BUILDDIR=$(shell pwd)/build OUTPUTDIR=$(shell pwd)/output ARCHIVE_SNAPSHOT=$(shell date -d "-1 day" +"%Y/%m/%d") -SOURCE_DATE_EPOCH=$(shell date -u -d "$(echo "$ARCHIVE_SNAPSHOT")" +"%s") +SOURCE_DATE_EPOCH=$(shell date -u -d "today 00:00:00" +"%s") .PHONY: clean clean: From 76713dc531772d385aabb231a863a737db6d8fcb Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Wed, 1 Apr 2026 13:46:20 +0200 Subject: [PATCH 22/28] Add user documentation about reproducing an image locally --- README.md | 9 ++- REPRO.md | 164 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+), 2 deletions(-) create mode 100644 REPRO.md diff --git a/README.md b/README.md index 72bc0dc..686bf12 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,8 @@ Arch Linux provides OCI-Compliant container images in multiple repositories: Four versions of the image are provided: `base` (approx. 150 MiB), `base-devel` (approx. 260 MiB) and `multilib-devel` (approx. 300MiB) containing the -respective meta package; and `repro` which is a bit for bit reproducible image -based on the `base` version (note that, to ensure reproducibility, the pacman keys +respective meta package; and `repro` which is a bit for bit reproducible version +of the `base` image (note that, to ensure reproducibility, the pacman keys are stripped from this image so you're expected to run `pacman-key --init && pacman-key --populate archlinux` before being able to update the system and install packages via `pacman`). @@ -74,6 +74,11 @@ There are multiple `make image-XXX` targets, where each creates the respective `archlinux:XXX` image based on the corresponding meta package. Currently those include `base`, `base-devel`, `multilib-devel` and `repro`. +### Reproducing the `repro` image + +To reproduce the `repro` image locally, follow the instructions +in [REPRO.md](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/blob/master/REPRO.md). + ## Pipeline ### Daily releases diff --git a/REPRO.md b/REPRO.md new file mode 100644 index 0000000..0c158bd --- /dev/null +++ b/REPRO.md @@ -0,0 +1,164 @@ +# Reproducing the `repro` image + +The `repro` image is a bit for bit [reproducible build](https://reproducible-builds.org/) +of the `base` image. +Note that, to ensure reproducibility, the pacman keys are stripped from this +image, so you're expected to run `pacman-key --init && pacman-key --populate archlinux` +before being able to update the system and install packages via `pacman`. + +To reproduce the `repro` image locally, follow the below instructions. + +## Dependencies + +Install the following Arch Linux packages: + +* make +* devtools (for the pacman.conf files) +* git (to fetch the commit/revision number) +* podman +* fakechroot +* fakeroot +* diffoscope (to optionally check the reproducibility of the rootFS) +* diffoci + +## Set required environment variables + +Prepare the build environment by setting the following environment variables: + +* IMAGE_BUILD_DATE: The build date of the `repro` image you want to reproduce. +For instance, if you're aiming to reproduce the `repro-20260331.0.508794` image: +`export IMAGE_BUILD_DATE="20260331"` +* IMAGE_BUILD_NUMBER: The build number of the `repro` image you want to reproduce. +For instance, if you're aiming to reproduce the `repro-20260331.0.508794` image: +`export IMAGE_BUILD_NUMBER="0.508794"` +* ARCHIVE_SNAPSHOT: The date of the Arch Linux repository archive snaphot to build +the image against. This is based on the `IMAGE_BUILD_DATE`: +`export ARCHIVE_SNAPSHOT=$(date -d "${IMAGE_BUILD_DATE} -1 day" +"%Y/%m/%d")` +* SOURCE_DATE_EPOCH: The value to normalize timestamps with during the build. +This is based on the `IMAGE_BUILD_DATE`: +`export SOURCE_DATE_EPOCH=$(date -u -d "${IMAGE_BUILD_DATE} 00:00:00" +"%s")` + +## Build the rootFS and generate the Dockerfile + +From a clone of the [archlinux-docker](https://gitlab.archlinux.org/archlinux/archlinux-docker) +repository, build the rootFS with the required paramaters: + +```bash +make \ + ARCHIVE_SNAPSHOT="$ARCHIVE_SNAPSHOT" \ + SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" \ + $PWD/output/Dockerfile.repro +``` + +The following built artifact will be located in `$PWD/output`: + +* repro.tar.zst (the rootFS) +* repro.tar.zst.SHA256 (sha256 hash of the rootFS) +* Dockerfile.repro (the generated Dockerfile) + +## Optional - Check the rootFS reproducibility + +At that point, if the above artifacts built for the image you're aiming to reproduce +are still available for download from the +[archlinux-docker pipelines](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/pipelines) +artifacts, you can optionally compare the content of the `repro.tar.zst.SHA256` +file from the pipeline to the one generated during the above local build (which +should be the same, indicating that the rootFS has been successfully reproduced). + +Additionally, you can check differences between the `repro.tar.zst` tarball from +the pipeline and the one built during your local build with `diffoscope`: +`diffoscope /tmp/repro.tar.zst $PWD/output/repro.tar.zst` (where `/tmp/repro.tar.zst` +is the rootFS tarball downloaded from the pipeline and `$PWD/output/repro.tar.zst` is +the rootFS tarball you just built. +This should show no difference, acting as additional indicator that the rootFS has been +successfully reproduced. + +If the artifacts have already expired from the +[archlinux-docker pipelines](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/pipelines) +artifacts, that's not a big deal. You are still able to check the reproducibility of the image itself. + +## Build the image + +You can now (re)build the image against the rootFS and Dockerfile generated in the previous step. +To do so, build the image with the required parameters: + +```bash +podman build \ + --no-cache \ + --source-date-epoch=$SOURCE_DATE_EPOCH \ + --rewrite-timestamp \ + -f "$PWD/output/Dockerfile.repro" \ + -t "archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}" \ + "$PWD/output" +``` + +The built image will be accessible in your local podman container storage under the name: +`localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}`. + +## Check the image reproducibility + +Pull the image you're aiming at reproducing from Docker Hub: +`podman pull docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}` + +Compare the digest of the image pulled from Docker Hub to the digest of the image you built +locally: + +```bash +podman inspect --format '{{.Digest}}' docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} +podman inspect --format '{{.Digest}}' localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} +``` + +Both digests should be identical, indicating that the image has been successfully reproduced. + +Additionally, you can check difference between the images pulled from Docker Hub and +the image you built with `diffoci`: +`diffoci diff --semantic --verbose podman://docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} podman://localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}` +This should show no difference, acting as additional indicator that the image has been +successfully reproduced (see the following section about the `--semantic` flag requirement). + +### Note about `diffoci` requiring the `--semantic` flag (a.k.a "non-strict" mode) + +Docker / Podman does not allow to have two images with the same name & tag combination stored +locally, [preventing them to be checked with `diffoci` as-is](https://github.com/reproducible-containers/diffoci/issues/74). +To work around this limitation, one of the two image has to be named differently, whether by +setting a different name / tag combination at build time or by renaming it post-build +with e.g. `podman tag`. + +However, the image name & tag combination is automatically reported (and updated in the case +of a renaming) in the image annotations / metadata and it's apparently not possible to fully overwrite +it during build or update it post-build in a straightforward way. This introduces unavoidable non-determinism +in the image annotations / metadata that `diffoci` will report by default. +See for instance the following `diffoci` output (with the reported difference being introduced by +using `podman tag` to "rename" one of the images with the "-rebuild" suffix, in order to avoid name collision): + +``` +Event: "DescriptorMismatch" (field "Annotations") + map[string]string{ + "io.containerd.image.name": strings.Join({ + "registry.archlinux.org/archlinux/archlinux-docker:repro-repro", +- "-orig", + }, ""), +- "org.opencontainers.image.ref.name": "repro-repro-orig", ++ "org.opencontainers.image.ref.name": "repro-repro", + } +``` + +Given that it's currently not possible to have two images with the same name & tag +combination stored locally and that it's also not possible to "normalize" the related +annotations / metadata during (or after) the build, we are not aware of a way to get a +fully successful `diffoci` output in default / strict mode (i.e., with *absolutely* no +reported differences). +This is why we are "forced" to run `diffoci` with the `--semantic` flag +([a.k.a "non-strict" mode](https://github.com/reproducible-containers/diffoci?tab=readme-ov-file#non-strict-aka-semantic-mode)), +which ignores some attributes, including image name annotations. + +While having to run `diffoci` with the `--semantic` flag (for the lack of another option) +just to workaround this image naming technical constraint is unfortunate, we can attest that: + +* This limitation is specific to metadata handling in container tooling and does not +affect the actual filesystem contents or runtime behavior of the image. +* The reported difference in the image name annotations is (or is supposed to be, at least) the **only** +difference being reported when comparing the two images. +* These image name annotations are not part of the hashed object when generating the image digest, +meaning that this difference does not go in the way of digest equality between the two images (allowing +us to claim bit for bit reproducibility regardless). From 4819df410a707654aa4e6c9c55adb39d54b8c2da Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Wed, 1 Apr 2026 14:11:41 +0200 Subject: [PATCH 23/28] Formatting and typo fixes in REPRO.md --- REPRO.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/REPRO.md b/REPRO.md index 0c158bd..6e2f2ca 100644 --- a/REPRO.md +++ b/REPRO.md @@ -27,21 +27,21 @@ Prepare the build environment by setting the following environment variables: * IMAGE_BUILD_DATE: The build date of the `repro` image you want to reproduce. For instance, if you're aiming to reproduce the `repro-20260331.0.508794` image: -`export IMAGE_BUILD_DATE="20260331"` + * `export IMAGE_BUILD_DATE="20260331"` * IMAGE_BUILD_NUMBER: The build number of the `repro` image you want to reproduce. For instance, if you're aiming to reproduce the `repro-20260331.0.508794` image: -`export IMAGE_BUILD_NUMBER="0.508794"` + * `export IMAGE_BUILD_NUMBER="0.508794"` * ARCHIVE_SNAPSHOT: The date of the Arch Linux repository archive snaphot to build the image against. This is based on the `IMAGE_BUILD_DATE`: -`export ARCHIVE_SNAPSHOT=$(date -d "${IMAGE_BUILD_DATE} -1 day" +"%Y/%m/%d")` + * `export ARCHIVE_SNAPSHOT=$(date -d "${IMAGE_BUILD_DATE} -1 day" +"%Y/%m/%d")` * SOURCE_DATE_EPOCH: The value to normalize timestamps with during the build. This is based on the `IMAGE_BUILD_DATE`: -`export SOURCE_DATE_EPOCH=$(date -u -d "${IMAGE_BUILD_DATE} 00:00:00" +"%s")` + * `export SOURCE_DATE_EPOCH=$(date -u -d "${IMAGE_BUILD_DATE} 00:00:00" +"%s")` ## Build the rootFS and generate the Dockerfile From a clone of the [archlinux-docker](https://gitlab.archlinux.org/archlinux/archlinux-docker) -repository, build the rootFS with the required paramaters: +repository, build the rootFS with the required parameters: ```bash make \ @@ -66,20 +66,16 @@ file from the pipeline to the one generated during the above local build (which should be the same, indicating that the rootFS has been successfully reproduced). Additionally, you can check differences between the `repro.tar.zst` tarball from -the pipeline and the one built during your local build with `diffoscope`: -`diffoscope /tmp/repro.tar.zst $PWD/output/repro.tar.zst` (where `/tmp/repro.tar.zst` +the pipeline and the one built during your local build with `diffoscope`: +`diffoscope /tmp/repro.tar.zst $PWD/output/repro.tar.zst` *(where `/tmp/repro.tar.zst` is the rootFS tarball downloaded from the pipeline and `$PWD/output/repro.tar.zst` is -the rootFS tarball you just built. +the rootFS tarball you just built)*. This should show no difference, acting as additional indicator that the rootFS has been successfully reproduced. -If the artifacts have already expired from the -[archlinux-docker pipelines](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/pipelines) -artifacts, that's not a big deal. You are still able to check the reproducibility of the image itself. - ## Build the image -You can now (re)build the image against the rootFS and Dockerfile generated in the previous step. +You can now (re)build the image against the rootFS and the Dockerfile generated in the previous step. To do so, build the image with the required parameters: ```bash @@ -110,11 +106,15 @@ podman inspect --format '{{.Digest}}' localhost/archlinux-docker:repro-${IMAGE_B Both digests should be identical, indicating that the image has been successfully reproduced. -Additionally, you can check difference between the images pulled from Docker Hub and -the image you built with `diffoci`: -`diffoci diff --semantic --verbose podman://docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} podman://localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}` +Additionally, you can check difference between the image pulled from Docker Hub and +the image you built locally with `diffoci`: + +```bash +diffoci diff --semantic --verbose podman://docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} podman://localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} +``` + This should show no difference, acting as additional indicator that the image has been -successfully reproduced (see the following section about the `--semantic` flag requirement). +successfully reproduced *(see the following section about the `--semantic` flag requirement)*. ### Note about `diffoci` requiring the `--semantic` flag (a.k.a "non-strict" mode) @@ -126,10 +126,11 @@ with e.g. `podman tag`. However, the image name & tag combination is automatically reported (and updated in the case of a renaming) in the image annotations / metadata and it's apparently not possible to fully overwrite -it during build or update it post-build in a straightforward way. This introduces unavoidable non-determinism +it during build or update it post-build in a straightforward way. +This introduces unavoidable non-determinism in the image annotations / metadata that `diffoci` will report by default. See for instance the following `diffoci` output (with the reported difference being introduced by -using `podman tag` to "rename" one of the images with the "-rebuild" suffix, in order to avoid name collision): +using `podman tag` to "rename" one of the images with the "-orig" suffix, in order to avoid name collision): ``` Event: "DescriptorMismatch" (field "Annotations") From 4dd4125eff6944a70f78c965538851de7deebea6 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Wed, 1 Apr 2026 15:34:16 +0200 Subject: [PATCH 24/28] Improve user doc in REPRO.md --- REPRO.md | 114 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 68 insertions(+), 46 deletions(-) diff --git a/REPRO.md b/REPRO.md index 6e2f2ca..e080380 100644 --- a/REPRO.md +++ b/REPRO.md @@ -21,27 +21,45 @@ Install the following Arch Linux packages: * diffoscope (to optionally check the reproducibility of the rootFS) * diffoci -## Set required environment variables +## Prepare the build environment Prepare the build environment by setting the following environment variables: -* IMAGE_BUILD_DATE: The build date of the `repro` image you want to reproduce. +* `BUILD_VERSION`: The build version of the `repro` image you want to reproduce. For instance, if you're aiming to reproduce the `repro-20260331.0.508794` image: - * `export IMAGE_BUILD_DATE="20260331"` -* IMAGE_BUILD_NUMBER: The build number of the `repro` image you want to reproduce. -For instance, if you're aiming to reproduce the `repro-20260331.0.508794` image: - * `export IMAGE_BUILD_NUMBER="0.508794"` -* ARCHIVE_SNAPSHOT: The date of the Arch Linux repository archive snaphot to build -the image against. This is based on the `IMAGE_BUILD_DATE`: - * `export ARCHIVE_SNAPSHOT=$(date -d "${IMAGE_BUILD_DATE} -1 day" +"%Y/%m/%d")` -* SOURCE_DATE_EPOCH: The value to normalize timestamps with during the build. -This is based on the `IMAGE_BUILD_DATE`: - * `export SOURCE_DATE_EPOCH=$(date -u -d "${IMAGE_BUILD_DATE} 00:00:00" +"%s")` + +```bash +export BUILD_VERSION="20260331.0.508794" +``` + +* `ARCHIVE_SNAPSHOT`: The date of the Arch Linux repository archive snaphot to build +the image against. This is based on the date included in the image's `BUILD_VERSION`: + +```bash +export ARCHIVE_SNAPSHOT=$(date -d "${BUILD_VERSION%%.*} -1 day" +"%Y/%m/%d") +``` + +* `SOURCE_DATE_EPOCH`: The value to normalize timestamps with during the build. +This is based on the date included in the image's `BUILD_VERSION`: + +```bash +export SOURCE_DATE_EPOCH=$(date -u -d "${BUILD_VERSION%%.*} 00:00:00" +"%s") +``` + +Then clone the [archlinux-docker](https://gitlab.archlinux.org/archlinux/archlinux-docker) +repository and move into it: + +```bash +git clone https://gitlab.archlinux.org/archlinux/archlinux-docker.git +cd archlinux-docker +``` + +Note that all the following instructions assumes that you are at the root of the +archlinux-docker repository cloned above. ## Build the rootFS and generate the Dockerfile -From a clone of the [archlinux-docker](https://gitlab.archlinux.org/archlinux/archlinux-docker) -repository, build the rootFS with the required parameters: +Build the rootFS with the required parameters: ```bash make \ @@ -58,18 +76,22 @@ The following built artifact will be located in `$PWD/output`: ## Optional - Check the rootFS reproducibility -At that point, if the above artifacts built for the image you're aiming to reproduce -are still available for download from the -[archlinux-docker pipelines](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/pipelines) -artifacts, you can optionally compare the content of the `repro.tar.zst.SHA256` +At that point, if the artifacts built for the image you're aiming to reproduce +are still available for download from the rootfs stage of the corresponding +[archlinux-docker pipeline](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/pipelines) +, you can optionally compare the content of the `repro.tar.zst.SHA256` file from the pipeline to the one generated during the above local build (which should be the same, indicating that the rootFS has been successfully reproduced). Additionally, you can check differences between the `repro.tar.zst` tarball from -the pipeline and the one built during your local build with `diffoscope`: -`diffoscope /tmp/repro.tar.zst $PWD/output/repro.tar.zst` *(where `/tmp/repro.tar.zst` -is the rootFS tarball downloaded from the pipeline and `$PWD/output/repro.tar.zst` is -the rootFS tarball you just built)*. +the pipeline and the one built during your local build with `diffoscope` +*(where `/tmp/repro.tar.zst` is the rootFS tarball downloaded from the pipeline and +`$PWD/output/repro.tar.zst` is the rootFS tarball you just built)*: + +```bash +diffoscope /tmp/repro.tar.zst $PWD/output/repro.tar.zst +``` + This should show no difference, acting as additional indicator that the rootFS has been successfully reproduced. @@ -84,24 +106,27 @@ podman build \ --source-date-epoch=$SOURCE_DATE_EPOCH \ --rewrite-timestamp \ -f "$PWD/output/Dockerfile.repro" \ - -t "archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}" \ + -t "archlinux:repro-$BUILD_VERSION" \ "$PWD/output" ``` The built image will be accessible in your local podman container storage under the name: -`localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}`. +`localhost/archlinux:repro-$BUILD_VERSION`. ## Check the image reproducibility Pull the image you're aiming at reproducing from Docker Hub: -`podman pull docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER}` + +```bash +podman pull docker.io/archlinux/archlinux:repro-$BUILD_VERSION +``` Compare the digest of the image pulled from Docker Hub to the digest of the image you built locally: ```bash -podman inspect --format '{{.Digest}}' docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} -podman inspect --format '{{.Digest}}' localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} +podman inspect --format '{{.Digest}}' docker.io/archlinux/archlinux:repro-$BUILD_VERSION +podman inspect --format '{{.Digest}}' localhost/archlinux:repro-$BUILD_VERSION ``` Both digests should be identical, indicating that the image has been successfully reproduced. @@ -110,7 +135,7 @@ Additionally, you can check difference between the image pulled from Docker Hub the image you built locally with `diffoci`: ```bash -diffoci diff --semantic --verbose podman://docker.io/archlinux/archlinux:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} podman://localhost/archlinux-docker:repro-${IMAGE_BUILD_DATE}.${IMAGE_BUILD_NUMBER} +diffoci diff --semantic --verbose podman://docker.io/archlinux/archlinux:repro-$BUILD_VERSION podman://localhost/archlinux:repro-$BUILD_VERSION ``` This should show no difference, acting as additional indicator that the image has been @@ -119,29 +144,26 @@ successfully reproduced *(see the following section about the `--semantic` flag ### Note about `diffoci` requiring the `--semantic` flag (a.k.a "non-strict" mode) Docker / Podman does not allow to have two images with the same name & tag combination stored -locally, [preventing them to be checked with `diffoci` as-is](https://github.com/reproducible-containers/diffoci/issues/74). +locally, [making it impossible to check two images with the same name with +`diffoci`](https://github.com/reproducible-containers/diffoci/issues/74) by cascade. To work around this limitation, one of the two image has to be named differently, whether by -setting a different name / tag combination at build time or by renaming it post-build -with e.g. `podman tag`. +setting a different name / tag combination at build time (as done in this guide) or by renaming +it post-build with e.g. `podman tag`. However, the image name & tag combination is automatically reported (and updated in the case of a renaming) in the image annotations / metadata and it's apparently not possible to fully overwrite it during build or update it post-build in a straightforward way. This introduces unavoidable non-determinism -in the image annotations / metadata that `diffoci` will report by default. -See for instance the following `diffoci` output (with the reported difference being introduced by -using `podman tag` to "rename" one of the images with the "-orig" suffix, in order to avoid name collision): +in the image annotations / metadata that `diffoci` will therefore systematically report by default. +See for instance the following `diffoci` output reporting a difference in the image name annotation: ``` Event: "DescriptorMismatch" (field "Annotations") map[string]string{ - "io.containerd.image.name": strings.Join({ - "registry.archlinux.org/archlinux/archlinux-docker:repro-repro", -- "-orig", - }, ""), -- "org.opencontainers.image.ref.name": "repro-repro-orig", -+ "org.opencontainers.image.ref.name": "repro-repro", - } + "io.containerd.image.name": strings.Join({ +- "docker.io/archlinux/archlinux:repro-20260331.0.508794", ++ "localhost/archlinux:repro-20260331.0.508794", + }, ""), ``` Given that it's currently not possible to have two images with the same name & tag @@ -154,12 +176,12 @@ This is why we are "forced" to run `diffoci` with the `--semantic` flag which ignores some attributes, including image name annotations. While having to run `diffoci` with the `--semantic` flag (for the lack of another option) -just to workaround this image naming technical constraint is unfortunate, we can attest that: +just to workaround this technical constraint is unfortunate, we can attest that: * This limitation is specific to metadata handling in container tooling and does not affect the actual filesystem contents or runtime behavior of the image. -* The reported difference in the image name annotations is (or is supposed to be, at least) the **only** -difference being reported when comparing the two images. -* These image name annotations are not part of the hashed object when generating the image digest, -meaning that this difference does not go in the way of digest equality between the two images (allowing +* The reported difference in the image name annotation when running `diffoci` in default / strict mode +is (or is supposed to be, at least) the **only** difference being reported when comparing the two images. +* This image name annotation is not part of the hashed object when generating the image digest, +meaning that this difference does not prevent digest equality between the two images (allowing us to claim bit for bit reproducibility regardless). From 93b7de821a983728c668543f1ad65d2cb97ac644 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Wed, 1 Apr 2026 19:19:26 +0200 Subject: [PATCH 25/28] Add disclaimer to REPRO.md --- REPRO.md | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/REPRO.md b/REPRO.md index e080380..982d5df 100644 --- a/REPRO.md +++ b/REPRO.md @@ -1,13 +1,33 @@ # Reproducing the `repro` image -The `repro` image is a bit for bit [reproducible build](https://reproducible-builds.org/) -of the `base` image. +The `repro` image provides a bit for bit [reproducible build](https://reproducible-builds.org) +of the `base` image. + Note that, to ensure reproducibility, the pacman keys are stripped from this image, so you're expected to run `pacman-key --init && pacman-key --populate archlinux` -before being able to update the system and install packages via `pacman`. +before being able to update the system and install packages via `pacman` when using this image. To reproduce the `repro` image locally, follow the below instructions. +## Disclaimer + +Reproducible builds [expect the same build environment across builds](https://reproducible-builds.org/docs/definition/). + +While it *should* be fine in most cases, this means we cannot guarantee that you will always be able +to successfully reproduce a specific image locally over time. + +Technically speaking, the older the image you're trying to reproduce is, the more chance there is +to have more or less significant differences between your build environment +and the one used to build the original image (for instance in terms of packages versions). +Such differences can affect the build (and the resulting artifacts). Please note that failing to +reproduce an image locally does not necessarily mean that it isn't reproducible per se, but can +just be the result of significant enough differences between your build environment and the one +used to build the original image. + +You can avoid (or mitigate) eventual issues due to such differences by restoring all packages +of your build environment to the build date of the original image (see the [related instructions +from the Arch Wiki](https://wiki.archlinux.org/title/Arch_Linux_Archive#Restore_all_packages_to_a_specific_date)). + ## Dependencies Install the following Arch Linux packages: From c0bb9639131a69cac66befe89ebdfdf84877cb26 Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Wed, 1 Apr 2026 20:02:28 +0200 Subject: [PATCH 26/28] Slight wording improvements --- README.md | 4 ++-- REPRO.md | 20 ++++++++++---------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 686bf12..6e0ab47 100644 --- a/README.md +++ b/README.md @@ -74,9 +74,9 @@ There are multiple `make image-XXX` targets, where each creates the respective `archlinux:XXX` image based on the corresponding meta package. Currently those include `base`, `base-devel`, `multilib-devel` and `repro`. -### Reproducing the `repro` image +### Reproducing a `repro` image -To reproduce the `repro` image locally, follow the instructions +To reproduce a `repro` image locally, follow the instructions in [REPRO.md](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/blob/master/REPRO.md). ## Pipeline diff --git a/REPRO.md b/REPRO.md index 982d5df..32e9826 100644 --- a/REPRO.md +++ b/REPRO.md @@ -1,4 +1,4 @@ -# Reproducing the `repro` image +# Reproducing a `repro` image locally The `repro` image provides a bit for bit [reproducible build](https://reproducible-builds.org) of the `base` image. @@ -11,7 +11,7 @@ To reproduce the `repro` image locally, follow the below instructions. ## Disclaimer -Reproducible builds [expect the same build environment across builds](https://reproducible-builds.org/docs/definition/). +Reproducible builds [expect the same build environment across builds](https://reproducible-builds.org/docs/definition/). While it *should* be fine in most cases, this means we cannot guarantee that you will always be able to successfully reproduce a specific image locally over time. @@ -33,8 +33,8 @@ from the Arch Wiki](https://wiki.archlinux.org/title/Arch_Linux_Archive#Restore_ Install the following Arch Linux packages: * make -* devtools (for the pacman.conf files) -* git (to fetch the commit/revision number) +* devtools +* git * podman * fakechroot * fakeroot @@ -74,7 +74,7 @@ git clone https://gitlab.archlinux.org/archlinux/archlinux-docker.git cd archlinux-docker ``` -Note that all the following instructions assumes that you are at the root of the +Note that all the following instructions assume that you are at the root of the archlinux-docker repository cloned above. ## Build the rootFS and generate the Dockerfile @@ -88,7 +88,7 @@ make \ $PWD/output/Dockerfile.repro ``` -The following built artifact will be located in `$PWD/output`: +The following resulting artifacts will be located in `$PWD/output`: * repro.tar.zst (the rootFS) * repro.tar.zst.SHA256 (sha256 hash of the rootFS) @@ -100,19 +100,19 @@ At that point, if the artifacts built for the image you're aiming to reproduce are still available for download from the rootfs stage of the corresponding [archlinux-docker pipeline](https://gitlab.archlinux.org/archlinux/archlinux-docker/-/pipelines) , you can optionally compare the content of the `repro.tar.zst.SHA256` -file from the pipeline to the one generated during the above local build (which +file from the pipeline to the one generated during your local build (which should be the same, indicating that the rootFS has been successfully reproduced). Additionally, you can check differences between the `repro.tar.zst` tarball from the pipeline and the one built during your local build with `diffoscope` *(where `/tmp/repro.tar.zst` is the rootFS tarball downloaded from the pipeline and -`$PWD/output/repro.tar.zst` is the rootFS tarball you just built)*: +`$PWD/output/repro.tar.zst` is the rootFS tarball built during your local build in the following example)*: ```bash diffoscope /tmp/repro.tar.zst $PWD/output/repro.tar.zst ``` -This should show no difference, acting as additional indicator that the rootFS has been +This should return no difference, acting as additional indicator that the rootFS has been successfully reproduced. ## Build the image @@ -135,7 +135,7 @@ The built image will be accessible in your local podman container storage under ## Check the image reproducibility -Pull the image you're aiming at reproducing from Docker Hub: +Pull the image you're aiming to reproduce from Docker Hub: ```bash podman pull docker.io/archlinux/archlinux:repro-$BUILD_VERSION From 4b15f9a1a11d11474cad201521962dc6192630ea Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Thu, 2 Apr 2026 09:24:16 +0200 Subject: [PATCH 27/28] Remove ldconfig auxiliary cache file only for the repro group --- Dockerfile.template | 2 +- scripts/make-dockerfile.sh | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Dockerfile.template b/Dockerfile.template index f7d3cf0..4bb2c65 100644 --- a/Dockerfile.template +++ b/Dockerfile.template @@ -38,7 +38,7 @@ COPY --from=verify /rootfs/ / RUN ldconfig && \ sed -i '/BUILD_ID/a VERSION_ID=TEMPLATE_VERSION_ID' /etc/os-release && \ - rm -f /var/cache/ldconfig/aux-cache + LDCONFIG_AUX_CACHE ENV LANG=C.UTF-8 CMD ["/usr/bin/bash"] diff --git a/scripts/make-dockerfile.sh b/scripts/make-dockerfile.sh index aec8e97..4def246 100755 --- a/scripts/make-dockerfile.sh +++ b/scripts/make-dockerfile.sh @@ -13,11 +13,13 @@ declare -rx SOURCE_DATE_EPOCH="$6" BUILD_VERSION="${BUILD_VERSION:-dev}" CI_COMMIT_SHA="${CI_COMMIT_SHA:-$(git rev-parse HEAD)}" -# Honor SOURCE_DATE_EPOCH for the repro GROUP +# Honor SOURCE_DATE_EPOCH and delete non-determistic ldconfig auxiliary cache file for the repro GROUP if [[ "$GROUP" == "repro" ]]; then CREATED_TIMESTAMP=$(date -u -d "@$SOURCE_DATE_EPOCH" +%Y-%m-%dT%H:%M:%SZ) + LDCONFIG_AUX_CACHE="rm -f /var/cache/ldconfig/aux-cache" else CREATED_TIMESTAMP=$(date -Is) + LDCONFIG_AUX_CACHE="true" fi sed -e "s|TEMPLATE_ROOTFS_FILE|$ROOTFS_FILE|" \ @@ -27,4 +29,5 @@ sed -e "s|TEMPLATE_ROOTFS_FILE|$ROOTFS_FILE|" \ -e "s|TEMPLATE_VERSION_ID|$BUILD_VERSION|" \ -e "s|TEMPLATE_REVISION|$CI_COMMIT_SHA|" \ -e "s|TEMPLATE_CREATED|$CREATED_TIMESTAMP|" \ + -e "s|LDCONFIG_AUX_CACHE|$LDCONFIG_AUX_CACHE|" \ Dockerfile.template > "$OUTPUTDIR/Dockerfile.$GROUP" From 96c00dc076aceccaf44883b5fc801a4ce1812aac Mon Sep 17 00:00:00 2001 From: Robin Candau Date: Thu, 9 Apr 2026 22:33:22 +0200 Subject: [PATCH 28/28] Link the 'diffoci' upstream issue about the 'image naming paradox' in repro documentation --- REPRO.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/REPRO.md b/REPRO.md index 32e9826..dc009ae 100644 --- a/REPRO.md +++ b/REPRO.md @@ -164,8 +164,8 @@ successfully reproduced *(see the following section about the `--semantic` flag ### Note about `diffoci` requiring the `--semantic` flag (a.k.a "non-strict" mode) Docker / Podman does not allow to have two images with the same name & tag combination stored -locally, [making it impossible to check two images with the same name with -`diffoci`](https://github.com/reproducible-containers/diffoci/issues/74) by cascade. +locally, [making it impossible to compare two images with the same name with +`diffoci`](https://github.com/reproducible-containers/diffoci/issues/74). To work around this limitation, one of the two image has to be named differently, whether by setting a different name / tag combination at build time (as done in this guide) or by renaming it post-build with e.g. `podman tag`. @@ -173,7 +173,7 @@ it post-build with e.g. `podman tag`. However, the image name & tag combination is automatically reported (and updated in the case of a renaming) in the image annotations / metadata and it's apparently not possible to fully overwrite it during build or update it post-build in a straightforward way. -This introduces unavoidable non-determinism +This introduces unavoidable differences in the image annotations / metadata that `diffoci` will therefore systematically report by default. See for instance the following `diffoci` output reporting a difference in the image name annotation: @@ -190,7 +190,7 @@ Given that it's currently not possible to have two images with the same name & t combination stored locally and that it's also not possible to "normalize" the related annotations / metadata during (or after) the build, we are not aware of a way to get a fully successful `diffoci` output in default / strict mode (i.e., with *absolutely* no -reported differences). +reported differences, see the [related upstream report](https://github.com/reproducible-containers/diffoci/issues/266)). This is why we are "forced" to run `diffoci` with the `--semantic` flag ([a.k.a "non-strict" mode](https://github.com/reproducible-containers/diffoci?tab=readme-ov-file#non-strict-aka-semantic-mode)), which ignores some attributes, including image name annotations.