I've cleaned up the cycle cracking in parallel_emerge to hopefully be
significantly easier to understand. This also fixes some bugs.
Most significantly, this patch restores the dependency cracking so that it
actually cracks cycles that are against the emerge ordering, instead of
cracking everything but those dependencies. This was only broken temporarily
due to my fix for Bug 5795. Oops. Fortunately, this bug doesn't affect many
people, because make_chroot --fast is deprecated anyway.
TEST=make_chroot --fast, parallel_emerge -pe hard-host-depends
BUG=chromium-os:5795
Review URL: http://codereview.chromium.org/3184011
Previously, parallel_emerge only broke counter-plan dependencies if they were
mutually cyclic. This doesn't work, because these deps might be involved in
an unrelated cycle.
To fix this, we break all counter-plan dependencies that are involved in any
cycle. This fixes make_chroot --fast, which currently fails with a deadlock.
Besides the above, I've also added additional cycle-checking to the install
plan generation code so that cyclic dependencies are caught earlier. I also
cleaned up FindCyclesAtNode to be a bit more understandable -- I wasn't 100% sure
that it was actually correct so I cleaned it up so that it was easier for me to
understand that it is in fact correct. Not sure that this part fixes any bugs,
but it's easier for me to analyze the behavior of FindCyclesAtNode now.
TEST=make_chroot --fast (Note that make_chroot --fast is deprecated, but it's
good to fix the cycle cracking anyway.)
BUG=chromium-os:5795
Review URL: http://codereview.chromium.org/3156018
When we pass on --jobs to individual emerge processes, it causes them
to hide their output. This makes it harder to debug why they are
failing or hanging. Example: http://goo.gl/C6Ru
TEST=Made sure that output is shown by emerge processes when --jobs is used.
BUG=chromium-os:5684
Review URL: http://codereview.chromium.org/3168012
This should cut several minutes off the time for build_packages --nousepkg
--showoutput.
TEST=Ran build_packages --showoutput, verified output.
BUG=chromium-os:5647
Review URL: http://codereview.chromium.org/3106008
By default, emerge will pause and beep when it has messages that it wants the
user to see. This beeping is not useful during a parallel merge, so we turn it
off.
TEST=Ran parallel_emerge udev. It stopped beeping and completed faster.
BUG=none
Review URL: http://codereview.chromium.org/3080005
BUG=none
TEST=run build_packages on --board=tegra2_dev-board-opengl right now, since binhost hasn't been populated yet.
Review URL: http://codereview.chromium.org/3035058
Also update parallel_emerge to print output in general if you hit
CTRL-C or if a job is running for more than an hour. Also cleanup exit
handling so that emerge exits a bit more cleanly, and so that we don't
hang if emerge throws an Exception.
BUG=none
TEST=Ran ./parallel_emerge --show-output -uDNve chromeos
Review URL: http://codereview.chromium.org/3010056
Currently, our build occasionally fails with errors about about symlinks
already existing. This usually only slows down the build, because we have
retries, but it would be good to fix the problem.
This patch updates parallel_emerge to use the no-env-update feature for cross-
compiles. This feature skips the env_update step, which isn't thread-safe, and
isn't necessary for when we're cross-compiling and don't actually need to run
the binaries. Instead we run the env_update at the end, avoiding a thread-safety
issue.
TEST=Ran full build --nousepkg with env_update occuring at end.
BUG=chromium-os:5382
Review URL: http://codereview.chromium.org/3034051
When --onlydeps is specified, we don't want to merge packages that were
specified on the command-line. Previously, it threw an exception in this
case because it couldn't find the command-line packages in the list of
installed packages.
TEST=Run ./parallel_emerge --onlydeps autotest, and make sure it
doesn't crash.
Run ./parallel_emerge --board=x86-generic -po chromeos power_manager and make sure it does emerge power_manager, and doesn't crash.
BUG=none
Review URL: http://codereview.chromium.org/2868094
The icedtea and o3d packages take a really long time to rebuild (over 30 mins
each), so we can give our developers a break if we exempt them from automatic
rebuilds.
TEST=Rev the portage ebuild or the fontconfig ebuild and run build_packages.
Without the change, takes over 45 mins. With the change, it takes 5 mins.
BUG=none
Review URL: http://codereview.chromium.org/3029051
Rather than using split, we can just use os.path.basename(path) to get
the username from the homedir name. This is a bit simpler.
I've also added a comment which I missed in my last commit.
TEST=Ran ./parallel_emerge, verified the new method of getting
PORTAGE_USERNAME gets the right username.
BUG=none
Review URL: http://codereview.chromium.org/3056002
I tried to disable parallel_emerge yesterday, but my change didn't work,
so the buildbots are still running into occasional related issues.
Portage has two flags for doing collision protection: collision-protect
and protect-owned. The protect-owned feature is enabled by default and
is quite useful: it checks to make sure that we don't have multiple
packages that own the same file. The collision-protect feature is more
strict, and less useful: it fails if it finds a conflicting file, even
if that file was created by an earlier ebuild that failed to install.
We want to disable collision-protect here because we don't handle
failures during the merge step very well -- we just leave the old
Unfortunately, we haven't quite figure out the best way to handle
these failures in parallel emerge, so for now we disable the flag.
TEST=Created a fake collision, made sure that packages still merge.
BUG=none
Review URL: http://codereview.chromium.org/2825076
collision-protect doesn't make sense for parallel_emerge, because
we don't lock around merges. Also, if merge fails part-way through,
it's strange to fail just because some files are lying around.
This fixes an issue with the buildbots.
TEST=Checked that collision-protect feature gets deleted by this.
BUG=none
Review URL: http://codereview.chromium.org/3061029
Our package graph is cyclic, so parallel_emerge needs to handle cycles
correctly in all cases. PrebuiltsReady should only need to check each package
once, so we should set cache[pkg] to True if we found the package in the cache.
TEST=Ran build_packages --fast
BUG=none
Review URL: http://codereview.chromium.org/3047009
Instead of loosely wrapping emerge, parallel_emerge now integrates tightly with emerge. This boosts performance while allowing us to map dependencies more accurately.
With this change, build_image --fast can clock under 4 minutes, and build_packages --fast now clocks as low as 4:30. With the --rebuild option, build_packages takes 5:30, but it has the big benefit of producing actually-correct results.
Note that this change also depends on us updating the various build scripts to prefix
calls to parallel_emerge with sudo -E.
TEST=Ran several parallel_emerge test cases, build_packages --fast, and
build_image --fast
BUG=none
Review URL: http://codereview.chromium.org/2891013
When we run custom actions in parallel_emerge, we should pass the list of
packages on to emerge. Otherwise emerge won't know what packages to unmerge.
Besides the above, also fix an issue where --workon doesn't override
--getbinpkgonly or --usepkgonly.
TEST=Ran parallel_emerge --unmerge dhcpd and parallel_emerge --usepkgonly --board=x86-generic
--workon=power_manager power_manager
BUG=none
Review URL: http://codereview.chromium.org/2927013
Currently, parallel_emerge doesn't know how to handle uninstall actions, so it
actually installs packages when it should be uninstalling them. This patch
fixes that.
TEST=sudo ./parallel_emerge -uDNv hard-host-depends, and make sure it
uninstalls nih-dbus-tool, which is blocked by hard-host-depends
BUG=none
Review URL: http://codereview.chromium.org/2990001
Packages specified as workon packages are always built from source.
Dependencies of workon packages are also built from source.
Dependencies are calculated with --selective=n so that workon packages will be included in the install list. Packages that are not being worked on and are being unnecessarily replaced because of --selective=n are filtered out later.
This patch also fixes a bug with world file updating -- previously the world file was never updated because the packages were getting removed by RemoveInstalledPackages. We now keep the appropriate world file updates, and instead update SanitizeTree to handle world file updates without trouble.
I also optimized the cycle cracking algorithm so that it only visits each node once. This seems to improve run time significantly on graphs that have cycles.
TEST=Ran ./parallel_emerge -uDNvpg --board=x86-generic chromeos --workon=chromeos-chrome &&
./parallel_emerge -uDNvpg --board=x86-generic chromeos --workon=libcros &&
./parallel_emerge -uDNvpg world --workon=libxml2 &&
./parallel_emerge -uDNv -p world hard-host-depends --workon='bcel'
BUG=none
Review URL: http://codereview.chromium.org/2959006
- Add support for --jobs flag.
- Add support for --depclean with no arguments.
- Cleanup comments.
- Print out error details the first time a package fails.
TEST=Ran full build_packages --fast
BUG=none
Review URL: http://codereview.chromium.org/2853031
- Unmerge appropriate packages during upgrades and downgrades.
- Calculate time spent in dependency generation to the tenth of a second.
- Only track dependencies of packages that are actually being installed.
- Ignore PDEPEND, as it has no impact on dependency ordering.
- Only break dependency chains that go against Portage's install order.
- Rename Failed -> Retrying.
- Print emerge command lines as they are run.
TEST=Emerged hard-host-depends and ran build_packages with parallel_emerge
BUG=none
Review URL: http://codereview.chromium.org/2886010
Include checked in parallel emerge,
with an optional (default false) argument
in build_image to turn it on.
Review URL: http://codereview.chromium.org/2827037
