This is some fallout from converting scripts from python2 to
python3. Output received from the functions in subprocess module now
return bytearrays, but we operate on them as if they were a text. So
decode the bytearrays to strings. Otherwise we are either getting some
junk values passed to the command line utilities (for example:
`b'/dev/loop2'` instead of `/dev/loop2`), or exceptions are thrown,
because a function expected a string.
This is just a conversion done by 2to3 with a manual updates of
shebangs to mention python3 explicitly. The fixups for bytearray vs
string issues will follow up.
To be able to support arm64 native SDK without cross builds, we should
make generate_au_zip support both architectures, amd64 and arm64.
Without doing that, `build_image` fails with `ERROR : Required
WHITE_LIST items ld-linux-x86-64.so.2 not found!!!`, because the
script recognizes only amd64 libs in WHITE_LIST.
We should first determine the architecture in build_image, before
running generate_au_zip, and pass the architecture, either amd64 or
arm64. Also add allow_list and ld_linux parameters to necessary
functions.
A number of places refer to these paths and that number is going to
grow. Since the standard pattern is to use environment variables for
commonly used paths it is time to add ones for these:
REPO_CACHE_DIR
REPO_MANIFESTS_DIR
Add core_upload_update to au-generator.zip which requires some extra
logic to make it runnable anywhere it may be. To organize the code a
little better all the delta_generator calls have been moved to
cros_generate_update_payload. core_upload_update is now just a wrapper
around cros_generate_update_payload and core-admin.
Cgpt was moved and a symlink based wrapper was added. That wrapper will
be improved soon, when when that's true we'll need to change this back.
A specific note... cgpt is currently statically linked. If that wrapper does
not remain statically linked, then a simple revert won't be enough.
BUG=chromium-os:39814
TEST=Manual au-generate.zip creation.
Change-Id: I2705b1eddd8ef28c7eb099512513daf80f586218
Reviewed-on: https://gerrit.chromium.org/gerrit/45128
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
We don't need to support POSIX shell, so convert to bash.
BUG=None
TEST=`cbuildbot {amd64,x86,arm}-generic-full chromiumos-sdk daisy-release` worked
Change-Id: I33ad25d2310c593f3e346d955e3aa27da41091fc
Reviewed-on: https://gerrit.chromium.org/gerrit/41271
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The new glibc caused a series of problems with Paygen based delta generation.
First, the new glibc is newer than what is used in workstations, or on
the Paygen servers. Since libc wasn't one of the libraries bundled up
by generate_au_zip.py, the executables bundled up (including delta_generator)
would fail because of unstatisfied library requirements at startup.
When the new libc was included by generate_au_zip.py, the delta_generator
executable started causing the dynamic loader to segfault during startup,
presumably because it was linked for a newer version. This means that
the new loader needs to be bundled and explicitly used when invoking
the new executables (thanks David James for figuring this out!).
Next, after including all of the new libraries, bash would crash at startup
with these libraries at the start of LD_LIBRARY_PATH. Since Paygen which
set LD_LIBRARY_PATH before invoking a shell script which invokes the delta
generator, this means delta generation would crash before it started.
If I modified Paygen to not set LD_LIBRARY_PATH, then older au_generator.zip
files could not be properly handled (and we do so regularly).
This change moves the required dynamic libraries into a subdir of the zip
file which will not be in Paygens LD_LIBRARY_PATH which allows bash to
operate correctly.
It also renames each dynamically linked executable from xxx to xxx.wrapped
and creates a shell script named xxx which invokes xxx.wrapped with the
new LD_LIBRARY_PATH and using ld-linux-x86-64.so.2 to do so.
This change also moves a number of constants from inline in various
functions to constants at the top of the script, and introduces a WHITE_LIST
to cause a build failure if ld-linux-x86-64.so.2 isn't present.
(Fixing chromium-os:32550 would help with that)
BUG=chromium-os:32542
TEST=Generated au_generator.zip by hand, extracted files in a test directory,
and generated a delta using Paygen's command line:
LD_LIBRARY_PATH=. PATH=.:$PATH ./cros_generate_update_payload --image ../chromiumos_test_image.bin --output fuzzy --outside_chroot
Change-Id: I90d18a6d17a8f9824b19a6ce480048e388832b56
Reviewed-on: https://gerrit.chromium.org/gerrit/27443
Reviewed-by: Jay Srinivasan <jaysri@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Fix the blacklisting logic, and re-add the magic library linux-vdso.so.
BUG=chromium-os:32542
TEST=Run by hand, .zip contents examined by hand.
Change-Id: I94d99bf62e5eb011ac70428d7cebeaa852519a78
Reviewed-on: https://gerrit.chromium.org/gerrit/27398
Reviewed-by: Jay Srinivasan <jaysri@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Commit-Ready: Don Garrett <dgarrett@chromium.org>
We were blacklisting some dependant libraries for the delta generator when
we bundle them all up together. When glibc in the chroot was update to be
newer than glibc on our workstations, this broke payload generation.
The real fix is to link delta_generator statically so we don't have to do
this goofy library bundling business (chromium-os:32544). In the mean time,
bundle everything up to get us working again.
BUG=chromium-os:32544
TEST=Trybot run, but not fully verified.
Change-Id: I7b7d247f4edd8ecbab772807f6d2c4e7fdfd414a
Reviewed-on: https://gerrit.chromium.org/gerrit/27327
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
We have updated the toolchain in the chroot to use a newer version of
libstdc++ than what is installed on our workstations. This means
this additional library is required to generate deltas outside of the
chroot.
BUG=chromium-os:25277
TEST=Generated zip file, and ran contents outside of chroot on my workstation.
Change-Id: I5a90ee355aabd4849a9186a9a66e3dc9b1c51d52
Reviewed-on: https://gerrit.chromium.org/gerrit/14597
Reviewed-by: Eric M. Blake <eblake@google.com>
Reviewed-by: Eric Blake <eblake@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Commit-Ready: Don Garrett <dgarrett@chromium.org>
A modest number of different scripts and files have a usage
restricted to build_image. Move those files to build_library,
to prevent them accumulating unwanted clients.
BUG=chromium-os:17390
TEST=build_image
TEST=grep relevant sources for references to all the files
Change-Id: I3e9f6447ec34c09ea2d61f29ac343386a1e122b9
Reviewed-on: http://gerrit.chromium.org/gerrit/4762
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
