This change adds a command line option to build_packages to stop
execution of the script after breaking circular dependencies (i.e.
before board packages are built). Helpful for debugging package build
issues.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change removes torcx libraries, references, and commandline options
from build automation scripts and from build_library/.
Containerd and docker are shipped via sysexts which are included in the
base image.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
That way we can see a report of what emerge is going to do and the
status of the use flags for the installed packages. The downside is
that we are going to have reports about using deprecated and
unsupported profile in even more places.
Emerge flags are cryptic in general, but short flags even more so, so
expand them. While at it, I noticed some places where bash arrays
could be used, so convert those places too.
We need to fix another symlink created by gcc-config, so extend the
function that was doing it for some other links and rename it - it's
not about only liblto any more.
Torcx manifest may contain paths and URLs as locations of
packages. There are two kinds of packages - vendored and
extra. Vendored packages normally have two locations - path to the
directory inside the image where the package is (which is why it's
called vendored), and a URL to the package on some remote
server. Extra packages only have a URL. But the URLs are added only
when we tell the build_torcx_store script to upload the packages at
the same time, which is what the old build pipeline was doing. With
the new pipeline, the upload happens as a separate step, thus the
upload is disabled when invoking build_torcx_store, and so the
packages are not getting URLs set. This change went unnoticed, because
a kola test checking the generated torcx manifest was only checking if
there is at least one location, either path or URL, and all the new
releases have no extra packages, only vendored ones.
When backporting the new pipeline to old LTS, the kola tests started
to fail, because old LTS had one extra package, and this is how I
noticed the problem.
In 9fba5789f9 we introduced
--torcx_output_root as an optional command line parameter
and had it default to "${DEFAULT_BUILD_ROOT}", inadvertently
diverging from the previous default, which was
"${DEFAULT_BUILD_ROOT}/torcx".
This change sets the correct default root "${DEFAULT_BUILD_ROOT}/torcx" to bring
build_packages back into alignment with build_image.
ci-automation builds on the SDK container and simplifies CI automation
build tasks (SDK bootstrap, SDK container, packages, image, VMs).
See ci-automation/README.md for a brief introduction.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
With the update of the sys-apps/util-linux package, another cycle has
reared its ugly head. Looks like the package got an optional
dependency on sys-fs/cryptsetup, so break it.
The scripts that invoked `cros_workon` without specifying a path to
the script were not calling the internal `cros_workon` directly, but
rather a copy installed in `/usr/bin/cros_workon`.
`/usr/bin/cros_workon` comes from the `coreos-base/cros-devutil` and
is a wrapper script that sources `common.sh` file to figure the
location of the `scripts` and finally invokes the internal
`cros_workon`. Curious thing is that the sourced `common.sh` comes
from the `/usr/lib/crosutils` directory and contents of the directory
come from the `dev-util/crosutils` package. And that `common.sh` is
different from the one in the scripts directory, but fortunately the
part that detects the path to the `scripts` directory is the same. I'm
not sure where where exactly the copy of `common.sh` in
`/usr/lib/crosutils` comes from - likely from somewhere in
`https://chromium.googlesource.com/chromiumos/platform/crosutils`.
Just cut the middle layers and call the internal copy of `cros_workon`
directly.
The `--jobs` parameter that some scripts defined was not used anywhere
in jenkins or mantle. So the value of the parameter always ended up
being equal to `${NUM_JOBS}` set by `common.sh`. Also, even if the
`--jobs` parameter was used for some script, that script usually
didn't forward the jobs value to other scripts, so the other scripts
ended up using `${NUM_JOBS}` again. Also, the `${FLAGS_jobs}` variable
was used by some functions in the build library, and those functions
were sometimes invoked by scripts that didn't define the
`${FLAGS_jobs}` variable. It is tedious to track which script should
actually define the parameter, and where it should be forwarded.
Just get rid of this half-working pretense. If you want to affect how
many jobs `emerge` uses, export the `NUM_JOBS` environment variable
before calling any script.
For `EMERGE_FLAGS` and `REBUILD_FLAGS` we unconditionally specify the
`--jobs` flag's value to `${NUM_JOBS}` because they are passed to
`emerge`. On the other hand we drop the `--jobs` parameter from the
`UPDATE_ARGS` variable, because this variable passed to `setup_board`
or `update_chroot`, which don't have this flag any more.
Previously we broke the cycle caused by sys-apps/util-linux only,
while disabling cryptsetup USE flag in systemd to avoid another
cycle. That worked before, because the follow-up merge of the rest of
packages built sys-fs/cryptsetup before sys-apps/systemd. After an
update, the new portage is ordering the builds in different way and
sys-apps/systemd ended up being built before sys-fs/cryptsetup and
that failed during the configure phase because of unmet dependencies.
Better build all the packages taking part in the loop (not counting
the virtual packages), so we become less reliant on the package build
ordering. It is going to take slightly more time as we build a couple
of packages more.
Instead of rebuilding just one package and maybe rebuilding others as
a fallout, force rebuilding all the mentioned packages. This makes the
build process a bit more robust against package build ordering
changes.
May be useful when breaking multiple dep loops that have some common
packages, so we build them all once.
When the ebuild file changed but not its version nor its cross-workon
commit, the binary package would be used. Also some dependencies are not
encoded to trigger a rebuild of depending packages.
Allow to exclude some binary packages so that we can be sure that they
are rebuilt.
The idea is that once the installed board packages have changed,
the sysroot that was used to build any existing torcx packages may
no longer be compatible. It will therefore run build_torcx_store,
passing relevant options to it, to build a new torcx store of
images with the current sysroot.
This adds --skip_torcx_store to use the previous behavior.
When --usepkgonly is mixed with the right update flags it will
re-install any binary packages that have a newer build. Since the full
set of SDK and board packages are rebuilt quite often this leads to
excessive reinstalling.
setup_qemu_static only needs to be called once, and can
be done early so that it is available. Move the call
of setup_qemu_static from build_packages to setup_board.
Signed-off-by: Geoff Levand <geoff@infradead.org>
Chrooting in the target non-native arch rootfs is required
for update-bootengine to be run correctly.
Signed-off-by: Andrej Rosano <andrej@inversepath.com>
Before attempting to do @preserved-rebuild to fix linked against old
libraries remove any packages that no longer have corresponding ebuilds,
making them impossible to rebuild. This uses `--depclean`'s secondary
meaning: `--unmerge` but only remove packages without dependencies.
Rebuilds packages that are linked against old libraries that have been
upgraded or removed from the system. Skipping this can lead to shared
library checks looking ok in the build root but then built images have
broken library dependencies.
The --rebuild-if-unbuilt emerge option will recompile packages if any
build-time dependencies changed. This can be a useful safety to make
sure applications get re-linked and so forth. However recent versions of
portage have grown better defaults for choosing when to rebuild so lets
turn the build_packages flag around and leave emerge to do its default
behavior and see if that is sufficiently safe.
Previously we attempted to speed up the first build of a board by
pulling in binary packages regardless of their use flags, assuming that
if the package was already built the loop would not be an issue. Usually
this was true but not always. Now things like util-linux will always get
compiled when setting up a board for the first time but at least it
won't be as likely to fail.
Sorry, this is just getting bad. We need to switch to initializing the
board root stage3 style, similar to how the SDK or most any
semi-complicated Gentoo install starts. Breaking loops while merging
into a clean root is just too complicated.
On what this does: util-linux now has a udev *and* a systemd use flag.
Since we use systemd they both are effectively the same and pull in the
systemd package. This adds support for disabling both flags during the
loop breaking procedure.
If util-linux has a binary package it will be used, but if that binary
package has +udev it will pull in systemd. systemd has a loop that needs
to be broken too so if the util-linux loop breaker doesn't also handle
the systemd one it all falls apart.
In short the comment above the loop breaker code noting that we can try
this until it gets wonky. Well, it is wonky and we need to re-do how
build_packages works as a result. This is just a temporary workaround
until we figure out a larger restructuring.
Using parallel_emerge has been disabled by default for all commands
except build_image for quite a while now, build_image kept it just
because it was still a bit faster than normal emerge. Keeping
parallel_emerge complicates future changes to build_image so it needs to
drop it entirely. Since that means nothing uses it by default we might
as well just rip out support for it entirely.
The extra "dependency check" doesn't appear to be all that useful, so
don't waste time on it. Remove some unused command line options for
selecting which packages to build. Use new board-packages ebuild which
currently just includes 'coreos' and 'coreos-dev' but later can include
extra things like 'python-oem' which doesn't land in the base image.
