flatcar-scripts/pkg_auto

Scripts for package update automation

A quick start from blank state:

• clone scripts repo and create worktrees for the old state and new state (and optionally for the branch with package automation scripts if they are not a part of the main branch yet):
  • git clone https://github.com/flatcar/scripts.git scripts/main
  • cd scripts/main
  • PKG_AUTO="${PWD}/pkg_auto"
  • git worktree add --branch weekly-updates ../weekly-updates origin/buildbot/weekly-portage-stable-package-updates-2024-09-23
• prepare for generating reports (create a directory, download necessary stuff, create config):
  • mkdir ../../weekly-updates
  • cd ../../weekly-updates
  • "${PKG_AUTO}/download_sdk_and_listings.sh" -s ../../scripts/main -x aux-cleanups aux
    • call "${PKG_AUTO}/download_sdk_and_listings.sh" -h to get help
  • "${PKG_AUTO}/generate_config.sh" -a aux -n weekly-updates -o main -r reports -s ../../scripts/main -x file,wd-cleanups config
    • call "${PKG_AUTO}/generate_config.sh" -h to get help
• generate the reports:
  • "${PKG_AUTO}/generate_reports.sh" -w wd config
  • if the command above fails, the reports directory (see the -r reports flag in the call to generate_config.sh above) will have some reports that may contain hints as to why the failure happened
    • the reports directory may also contain files like warnings or manual-work-needed
    • the items in warnings file should be addressed and the report generation should be rerun, see below
    • the items in manual-work-needed are things to be done while processing the updates
• in order to rerun the report generation, stuff from previous run should be removed beforehand:
  • source wd-cleanups
  • rm -rf reports
  • "${PKG_AUTO}/generate_reports.sh" -w wd config
• if generating reports succeeded, process the updates, update the PR with the changelogs and update summaries:
  • this is the manual part, described below
• after everything is done (like the PR got merged), things needs cleaning up:
  • source wd-cleanups
  • rm -rf reports
  • source aux-cleanups

Processing the updates (the manual part)

The generated directory with reports will contain the updates directory. Within there are two files: summary_stubs and changelog_stubs. The rest of the entries are categories and packages that were updated. The first file, summary_stubs, contains a list of packages that have changed and TODO items associated to each package. It is mostly used for being pasted into the pull request description as an aid for the reviewers. The latter, changelog_stubs, can serve as a base for changelog that should be added to the scripts repo.

For each package in the summary_stubs there are TODO items. These are basically of four kinds:

• to review the changes in the ebuild
• to review the changes not in the ebuild (metadata, patch files)
• to review the occurences of the package name in the scripts repository
• to add a link to the release notes in case of a package update

It is possible that none of the changes in the package are relevant to Flatcar (like when a package got stabilized for hppa, for instance), then the package should be just dropped from the summary_stubs. Note that the package update is relevant, so as such should stay in the file.

The entries in changelog_stubs should be reviewed about relevancy (minor SDK-only packages should likely be dropped, they are seldom of interest to end-users) and the remaining entries should be updated with proper links to release notes.

There may be also entries in manual-work-needed which may need addressing. Most often the reason is that the new package was added, or an existing package stopped being pulled in. This would need adding an entry to the summary_stubs.

Another thing that to do is to check the security reports. If the updated package brings a fix for a security issue, it should be mentioned in the summary and an entry in a separate security changelog should be added.

Other scripts

There are other scripts in this directory. inside_sdk_container.sh is a script executed by generate_reports.sh inside the SDK to collect the package information. sync_packages.sh is a script that updates packages and saves the result to a new branch. update_packages.sh is sync_packages.sh + generate_reports.sh.