Commit Graph

23 Commits

Author SHA1 Message Date
David James
0614c4a11c Switch from gsdview.appspot.com to commondatastorage.googleapis.com.
This server requires that unsafe characters are quoted in URLs, so we do that
here.

Change-Id: I6b7c430671b02a8da11777029b0fdb59a87dcae1

BUG=chromium-os:9902
TEST=Ran unit tests. Ran example buildbot run and diff'd new version and old
     version of Packages file.

Review URL: http://codereview.chromium.org/5542002
2010-12-07 13:39:16 -08:00
David James
a8ae0bc63b Update prebuilt.py to set permissions correctly on created Packages file.
Since the local Packages file is a temporary file, other users won't have
permissions to read it. rsync copies those permissions over to the remote
server. To ensure remote server has the right permissions, we need to use
chmod a+r on either the local file or the remote file. In this case, I chose
to run rsync on the remote file.

TEST=rsync -av --chmod=a+r foo bar and verify permissions are added to bar.
BUG=chromium-os:5311

Change-Id: Iee4c8df00b36b9f9434546603195ea1906b289f1

Review URL: http://codereview.chromium.org/5501001
2010-12-01 13:43:48 -08:00
David James
74f0f17ad8 Use regular string ops, instead of urlparse.urljoin.
urlparse.urljoin('scheme:foo/bar/', 'baz') == 'baz', so it doesn't work the
way I expected. Instead, let's use regular string joins.

BUG=chromium-os:5311
TEST=Ran buildbot in test mode.

Change-Id: Iea4205573db602e0cc20af24a77d255d17e79221

Review URL: http://codereview.chromium.org/5357012
2010-11-30 13:56:52 -08:00
David James
63c51c4d3c Update cbuildbot.py and prebuilt.py to deduplicate preflight prebuilts.
Preflight prebuilts reference the last full build, so they should always be complete as long as the last full build is still there.

Also add host prebuilts to the preflight prebuilts.

BUG=chromium-os:5311
TEST=Run unit tests for cbuildbot and prebuilt.py. Test runs of cbuildbot.py with --dryrun.

Review URL: http://codereview.chromium.org/5344002

Change-Id: Id95f94c02cc2f6cbd70a029d4f8b94617b7cc071
2010-11-30 11:45:11 -08:00
David James
c509a906da Update cbuildbot.py to upload prebuilts from preflight buildbot.
This change updates cbuildbot.py to upload prebuilts and reference them via PREFLIGHT_BINHOST.

Also ensure make.conf ends in a newline, as it looks nicer.

BUG=chromium-os:5311
TEST=Test that prebuilt.py updates PREFLIGHT_BINHOST in right make.conf
     file. Test that cbuildbot.py runs prebuilt.py with right
     arguments. Run unit tests for cbuildbot and prebuilt.py. Test runs of cbuildbot.py with --dryrun. Test upload of Packages file to local web server and check that emerge works with them.

Change-Id: Iad03c6c469e05b9ee1cceff69cbe2bdd51225e25

Review URL: http://codereview.chromium.org/4969003
2010-11-30 11:36:57 -08:00
Scott Zawalski
b593c39e9d Add a RevGitWithRetry method to retry repo sync and git push
Modify repo sync to just sync the repo we are in,
this is what preflight does and is a proven acceptable approach over
the complete 'repo sync' approach.

BUG=8987
TEST=Ran retry command over dummy directories to be sure it retries
as expected.

Review URL: http://codereview.chromium.org/5082002
2010-11-16 10:00:07 -08:00
Chris Sosa
170bde0fd4 Fix for bug 8764 that breaks prebuilts for files with more than just keyword pairs.
I've also modified the unit test so it actually tests for this case.

Change-Id: I6688d3be7bf1d8962031fcf1d684ddcaab43ba9f

BUG=8764
TEST=Unittests

Review URL: http://codereview.chromium.org/4566001
2010-11-04 20:49:30 -07:00
Olof Johansson
99b3f551d8 prebuilt.py: fix variant detection regex
Without this, I get:

Traceback (most recent call last):
  File "./prebuilt.py", line 495, in <module>
    main()
  File "./prebuilt.py", line 491, in main
    git_sync=options.git_sync)
  File "./prebuilt.py", line 408, in UploadPrebuilt
    git_file = os.path.join(build_path, DetermineMakeConfFile(board))
  File "./prebuilt.py", line 378, in DetermineMakeConfFile
    raise UnknownBoardFormat('Unknown format: %s' % target)
__main__.UnknownBoardFormat: Unknown format: tegra2_seaboard

Change-Id: I1d595b7e1c596139acca344f7a145143686dace1

BUG=chromium-os:8388
TEST=./prebuilt.py  -p ~/chromiumos -b tegra2_seaboard  -u user@workstation:/tmp

Review URL: http://codereview.chromium.org/4184007
2010-10-29 09:50:23 -05:00
David James
0a4d1665fd Strip newlines from each line in make.conf.
This prevents us from having double newlines.

Change-Id: I9e9f9e9164872419bf263b15d2a7d048e98a0098

BUG=chromium-os:4843
TEST=Ran unit tests.

Review URL: http://codereview.chromium.org/4179002
2010-10-27 11:39:11 -07:00
David James
59b403d950 Add support for prebuilt uploading using rsync.
BUG=not yet
TEST=Try uploading prebuilts to codf30.jail. Also run unit tests.

Change-Id: Ib218e705633febeddbe5c104f67fa580e1fc7de4

Review URL: http://codereview.chromium.org/4102001
2010-10-25 16:41:50 -07:00
Anush Elangovan
01e1342394 Fix host prebuilts
Change-Id: I23b39dd422e902f6752c533eb55ac8baf508f4e4

BUG=
TEST=

Review URL: http://codereview.chromium.org/4124001
2010-10-25 15:10:56 -07:00
Scott Zawalski
e2dae7e751 Change message with file reference to filename 2010-10-25 10:35:44 -07:00
Scott Zawalski
d367ed7a85 Update prebuilt.py to use gsdview instead of commondatastorage 2010-10-21 16:21:43 -07:00
David James
4f7f855728 Update prebuilt.py to filter Packages database.
This ensures that the Packages database doesn't contain private packages.
This is necessary so that the database matches up with the actual files
we upload.

BUG=chromium-os:4843
TEST=Ran unit test

Change-Id: I0355b4ab867a0d08396e45b04523a487951475f4

Review URL: http://codereview.chromium.org/3930001
2010-10-20 14:48:04 -07:00
Scott Zawalski
fe3805ac16 Update prebuilt to modify make.conf files for the board/host targets.
This changes the format that I was updating from 'key value' to 'key=value'.
I modified my unittests to test this.

BUG=NA
TEST=Branch runs without commit.

Review URL: http://codereview.chromium.org/3858003
2010-10-18 16:36:44 -07:00
Scott Zawalski
96619e72e1 Update RemoteUpload to return a set Update if failed_uploads to check deeper
in the returned set to see if it is a bonified failure

BUG=NA
TEST=Local run to verify

Review URL: http://codereview.chromium.org/3667005
2010-10-11 15:31:53 -07:00
Scott Zawalski
e69a464192 Add an absolute path to gsutil that we will be using.
Change the way uploads work so we error if one doesn't sync properly.
2010-10-07 12:53:32 -07:00
Scott Zawalski
8fcab50063 Fix doc string and use self.assertRaises 2010-10-05 10:39:51 -07:00
Scott Zawalski
f5948251b6 Change prebuilt filters to use the private ebuilds to generate a
definitive list.
Add unittests to test an empty filter list, loading filters, and
filtering specific packages based off of mock filters.
2010-10-04 15:15:44 -07:00
Scott Zawalski
f9923b0164 Changed FILTER_PACKAGES to _FILTER_PACKAGES
Fixed a missed TODO item without username
2010-10-01 10:58:55 -07:00
Scott Zawalski
6332272fa7 A few more style changes and logic changes fro code review 2010-09-30 08:26:04 -07:00
Scott Zawalski
6f0584cd2d Update prebuilt and prebuilt_unittest to address comments in code
review.
2010-09-28 14:55:36 -07:00
Scott Zawalski
4f33955508 First submission of prebuilt upload script.
This introduces a way to upload prebuilts from the command line.

The following features are available

  * Multiprocess upload (cuts time down by a 1/5th for uploading)
  * Versioned uploading with git file commital
  * Host prebuilt uploading
  * Board prebuilt uploading

BUG=chromium-os:4843,chromium-os:5855

TEST=./prebuilt_unittest.py
{'/b/cbuild/build/chroot/build/x86-dogfood/packages/x11-misc/util-macros-1.5.0.tbz2':
'gs://chromeos-prebuilt/host/version/packages/x11-misc/util-macros-1.5.0.tbz2',
'/b/cbuild/build/chroot/build/x86-dogf}
.FILTERING /usr/local/package/oob
FILTERING /var/tmp/bibby.file
.....Updating stage 20100309/stage3-amd64-20100309.tar.bz2 to stage
test_update
.
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
2010-09-28 09:38:03 -07:00