Commit Graph

32 Commits

Author SHA1 Message Date
Krzesimir Nowak
45faa790cc pkg-auto: Fix occurences generation
Occurences file shows where the package name shows up in the
repository. It tries to be smart, so that checking for sys-devel/gcc
will not be showing sys-devel/gcc-config. But the smart check was
flawed as it ignored the forms like sys-devel/gcc-${PV}. Noticed when
trying to check occurences for sys-libs/libsepol and there were not
enough occurences shown.
2025-05-09 09:24:10 +02:00
Krzesimir Nowak
4bb175152f pkg-auto: Mark in-rootfs sysexts as sysexts in reports
They were showing up as "docker" or "containerd" in the reports, which
is confusing. "sysext-docker" or "sysext-containerd" makes it clear.
2025-04-30 12:45:01 +02:00
Krzesimir Nowak
1039ed0571
pkg-auto: Simplify the use of sed
Co-authored-by: James Le Cuirot <jlecuirot@microsoft.com>
2025-04-29 14:07:35 +02:00
Krzesimir Nowak
a8922be627
pkg-auto: Simplify error checking in emerge output
Co-authored-by: James Le Cuirot <jlecuirot@microsoft.com>
2025-04-29 14:07:13 +02:00
Krzesimir Nowak
ecd24bd5eb pkg-auto: Fix a shellcheck warning 2025-04-29 09:43:21 +02:00
Krzesimir Nowak
0d06b737ac pkg-auto: Disable shellcheck reference warnings
Two warnings, SC2034 and SC2178, pop up very often with the references
- shellcheck handles them poorly and produces a ton of bogus warnings
about them. Silence the warnings and drop most of the "shellcheck
disable" clauses.
2025-04-29 09:43:21 +02:00
Krzesimir Nowak
296efcdb22 pkg-auto: Add a setup for shellcheck's "source" checking
That way shellcheck sources some prepared files and learns about some
variables the sourced files define. Thanks to that, we can remove some
of the "shellcheck disable" clauses.
2025-04-29 09:43:21 +02:00
Krzesimir Nowak
13e84333b8 pkg-auto: Add md5-metadata cache diff reports to package reports 2025-04-29 09:43:21 +02:00
Krzesimir Nowak
016f35d8f5 pkg-auto: Add the md5-metadata cache diff library 2025-04-29 09:43:21 +02:00
Krzesimir Nowak
3b11e2404a pkg-auto: Add the highest-score-common-subsequence algorithm library
It's a more general variant of the longest-common-subsequence
algorithm.
2025-04-29 09:43:21 +02:00
Krzesimir Nowak
3233f6eafe pkg-auto: Add a library for parsing md5-metadata cache files 2025-04-29 09:43:21 +02:00
Krzesimir Nowak
7c9c2dfea4 pkg-auto: Fix some reference variable names
I'm trying to follow a convention where reference variable names end
with "_ref".
2025-04-29 09:43:21 +02:00
Krzesimir Nowak
e1a4d8e5a9 pkg-auto: Move debugging stuff to a separate file
I'll use it also in new libraries.
2025-04-29 09:43:21 +02:00
Krzesimir Nowak
fff6bd78b2 pkg-auto: Move regular expression for package name to gentoo_ver
I did it initially, because I think I'll use this expression in other
place too. In the end I didn't but I still think it's a better place
for it.
2025-04-29 09:43:21 +02:00
Krzesimir Nowak
80d12ea75f pkg-auto: Rework reports generation
Reports generation used to be executed four times. The number of runs
was a result of cartesian product of two sets - old and new state, and
of amd64 and arm64 architectures. It was pretty much a slow process
because egencache was called implicitly four times, and it was running
in a single-threaded fashion, and also SDK reports were duplicated
(they were the same for old-amd64 and old-arm64, and the same for
new-amd64 and new-arm64 runs).

This changes the generation, so it is being run only two times - once
for old state and once for new state. Every run generates SDK packages
reports and per-architecture board packages reports. Egencache will
now utilize more threads too.
2025-04-29 09:43:21 +02:00
Krzesimir Nowak
00572476c4 pkg-auto: Simplify SDK image override handling
There used to be a possibility to override used SDK image per
architecture, but the need for it disappeared once SDK images started
to contain the initial form of board rootfs for both amd64 and
arm64. This eliminated problems with cyclic dependencies errors
popping up while gather the package reports. So with this change it is
now only possible to specify just one SDK image to use for any
arch. This feature is not used all that often anyway.
2025-04-29 09:43:20 +02:00
Krzesimir Nowak
7b5841c039 pkg-auto: Add md5-metadata cache to reports
This adds an explicit generation of md5-metadata cache before any we
do any emerge invocations. That way we can have a copy of reports even
if emerge fails for some reason. But mostly the reason for this
copying is to consume the data later, outside the SDK container.
2025-04-29 09:43:20 +02:00
Krzesimir Nowak
b52676a64b pkg-auto: Add function for declaring structs
Declaring structs differs a bit from declaring typical variables in
that it takes one initializer and applies it to all the declared
variables.

Will be used a lot by upcoming libraries.
2025-04-29 09:43:20 +02:00
Krzesimir Nowak
3931cbff5f pkg-auto: Add a global variable name generator function
Some upcoming libraries will use this for their global variables. The
function is using a single counter, which ensures that the generated
names will be globally unique.
2025-04-29 09:43:20 +02:00
Krzesimir Nowak
cfd6a4057b pkg-auto: Move sets_split to util.sh
This will be used in other places, so make it available without
importing the big pkg_auto_lib.sh file.
2025-04-29 09:43:20 +02:00
Krzesimir Nowak
a75a5ed7bf pkg-auto: Strip only one space when printing help 2025-02-20 11:39:37 +01:00
Krzesimir Nowak
0ba6b62bf0 pkg-auto: Disable pager when printing package occurences 2025-02-06 13:45:28 +01:00
Krzesimir Nowak
020f9fe19c pkg-auto: Add some tools
These tools I found useful when I had to investigate why report
generation failed. Since report generation failed, I had no reports
about ebuild diffs or package occurences. The new scripts,
`diff_pkg.sh` and `occurences.sh`, allowed me to get these reports for
the troublesome package.
2025-01-22 13:32:39 +01:00
Krzesimir Nowak
6536b2cf3d pkg-auto: Emerge dev-lang/rust for SDK reports
The coreos-devel/sdk-depends is a metapackage that is emerged in
stage4 of the catalyst SDK build. But the fsscript that is being run
at the end of the stage4 build also pulls in dev-lang/rust package.
Thus pull both packages for the report.

This should result in updated dev-lang/rust to appear in the reports.
2025-01-22 13:16:06 +01:00
Krzesimir Nowak
9fe7006812 pkg-auto: Allow emerging multiple packages for reports 2025-01-22 13:16:06 +01:00
Krzesimir Nowak
310a8716f4 pkg-auto: Fix syncing eclass files
eclass is a file, as opposed to the package, which is a directory. So
doing `git -C eclass/foo.eclass log -- .` will fail because we can't
do "cd eclass/foo.eclass", which is what `-C` is trying to do.
2025-01-22 13:16:06 +01:00
Krzesimir Nowak
7d72d5d8a0 pkg-auto: Update README 2024-12-03 15:04:45 +01:00
Krzesimir Nowak
04b78d9657 pkg-auto: Address shellcheck complaints 2024-11-27 17:17:58 +01:00
Krzesimir Nowak
4eba9bec28 pkg-auto: Add a missing function 2024-11-27 16:49:41 +01:00
Krzesimir Nowak
ae67b87fba pkg-auto: Move some implementation detail down the file
I try to keep "public API" at the top of the file.
2024-11-27 16:00:58 +01:00
Krzesimir Nowak
dd09caba17 pkg-auto: Import a stripped-down version of eapi7-ver.eclass into impl
The eclass was removed from Gentoo, so we followed suit. This broke
the pkg-auto code. Thus I imported the eclass into the impl directory
as gentoo_ver.sh, threw away all the unnecessary parts and moved some
from pkg_auto_lib.sh to the new file.

This allowed me to also drop a hack where I was grepping for the
version regexp in the eclass. Now I'm just exporting it.
2024-11-27 16:00:58 +01:00
Krzesimir Nowak
4c5e550055 pkg-auto: Add package automation scripts
This adds some scripts I have been using for over a year to deal with
the weekly package updates.

It comes with a `README.md` which describes a workflow similar to my
own.

The `sync_packages.sh` and `update_packages.sh` scripts are currently
not used anywhere. The idea behind them was to use them for Github
Action, but that will come as a follow-up PR.
2024-11-27 16:00:57 +01:00