Commit Graph

79 Commits

Author SHA1 Message Date
Don Garrett
38a66574b6 Add more cbuild options, revise configs
BUG=chromium-os:11229
TEST=Local, unittests

Review URL: http://codereview.chromium.org/6245013

Change-Id: I103d693a2e6ce033c1f495f9f48450156ad72e94
2011-01-27 14:04:15 -08:00
Don Garrett
91c39378e0 Add buildbot functionality to cbuildbot.
Change-Id: I1291d2f52122fe345c5927a7ef27e20d07e917d7

BUG=chromium-os:11229
TEST=(building locally, still in progress)

Review URL: http://codereview.chromium.org/6317014
2011-01-26 11:53:26 -08:00
Chris Sosa
f58bc389d9 Create package keywords directory if it doesn't exist.
Clobbering the builder uncovered a bug that was lurking.

Change-Id: Ieb487d2112a7048438943aa96fba23e9c8412a13

BUG=chromium-os:8693
TEST=Running now

Review URL: http://codereview.chromium.org/6050004
2011-01-10 10:58:58 -08:00
Chris Sosa
efbce1e261 Add simple update test to pfq.
This CL also fixes the outstanding issue where to won't report progress in
image_to_live on updates.

Change-Id: I1efaf17f0fd5ebb367ae0872377b4d5d0bf5dbf6

BUG=chromium-os:8680,chromium-os:8901
TEST=Ran it with --debug on the x86 pfq locally.

Review URL: http://codereview.chromium.org/6135002
2011-01-10 09:56:34 -08:00
Chris Sosa
b1d17e24eb Move git config till after we sync.
This deals with the condition where on a clobber we don't actually
re-config to use ssh:// until the second run (because git config is
called before we first sync the code).

Change-Id: Icf1a46aced633b570db9d280b10ae6866af04709

BUG=chromium-os:10545
TEST=Unit tests

Review URL: http://codereview.chromium.org/6027006
2010-12-22 14:10:10 -08:00
Chris Sosa
6b69ba7121 Always run build_packages regardless of sync for chrome builder.
Change-Id: If08f3a0c1212e3e7d4ae8dc9f8aa92e2454d4d74

BUG=chromium-os:8693
TEST=Ran it

Review URL: http://codereview.chromium.org/6016005
2010-12-20 11:40:29 -08:00
Chris Sosa
d523d23f2e Remove explicit call to build Chrome.
Since the package.keywords change, it's already built as part of build_packages.
With this change we don't have to needlessly build chrome twice as part of the
pfq.

Change-Id: I8dae745496a5d6f6213663b034ffe8d279d4efd3

BUG=chromium-os:8693
TEST=Ran unittests

Review URL: http://codereview.chromium.org/5973003
2010-12-17 12:35:34 -08:00
Chris Sosa
7c3fdc13b0 Push prebuilts by default.
Change-Id: I0d03b037253f299e36c3621ff6e5970e9af2bb11

BUG=chromium-os:10375
TEST=NA

Review URL: http://codereview.chromium.org/5876002
2010-12-15 17:42:44 -08:00
Chris Sosa
d75d3b43c8 Undo all the manifest logic from cbuildbot.
We also print to /dev/stderr directly to avoid flushing issues which have been occassionally given us broken manifests printed out

Change-Id: I97819bbc31142432aa5ef926f99f2c4f90fd50ef

BUG=chromium-os:9714
TEST=Ran cbuildbot through sync

Review URL: http://codereview.chromium.org/5888003
2010-12-15 17:42:07 -08:00
Doug Anderson
9b0127c6f5 Fixed build failure w/ new sudo
Change-Id: I2760e33e1473ce04f7adeda39d0db5725eaa81ae

BUG=chromium-os:7072
TEST=Replaced ./build_packages with bash and ran command; valided env var set right.

Review URL: http://codereview.chromium.org/5930002
2010-12-15 16:51:49 -08:00
Chris Sosa
cd2196d6c5 Add package.keywords logic to build version of chrome we specify.
Change-Id: I32c17e7353b420d7a895acffab4579c9b861095c

BUG=chromium-os:8693
TEST=Ran it with chrome_rev=tot, latest_release and stable_release

Review URL: http://codereview.chromium.org/5854001
2010-12-14 17:57:02 -08:00
David McMahon
48db5d2068 Moving chromeos_version.sh to chromiumos_overlay.
BUG=chromium-os:9714
TEST=

Review URL: http://codereview.chromium.org/5513009

Change-Id: Ib569fe59f4e14551b198568b24014208b5d65647
2010-12-14 16:19:24 -08:00
Chris Sosa
97432e7e9e Don't push manifests for chrome pfq.
Change-Id: I252616725936b2c471026d39d06c3fbb2fa1ed65

BUG=chromium-os:8693
TEST=Ran with chrome_rev=tot

Review URL: http://codereview.chromium.org/5744003
2010-12-13 16:20:51 -08:00
Chris Sosa
d8a0a57c30 Add support to pushing unstable changes for the chrome pfq.
This specifically gets the pfq one step closer to being completely ready to rev.  We add --noprebuilts for the chrome pfq as its not ready to host prebuilts and instead we push unstable ebuilds.  Anush noticed an Anush today where we are revving even when we don't change, and I've addressed this also in this CL.

Specifically:

1)  Sort in cbuildbot long options by long format
2)  Add --noprebuilts so that the Chrome PFQ can use this for now.
3)  Fixed a unittest that had broken in a previous CL for cros_mark_chrome_as_stable.
4)  Add ability for chrome to only rev if something really has changed:

 -- This means, if the new ebuild either:
  -- doesn't have the exact same chrome_version
  -- or diff -Bu (actual diff) between last ebuild and new ebuild isn't the same (i.e. detecting a 9999 change or different CROS_SVN_COMMIT)

Change-Id: I9b289c2168d6868299573f5d7c7a676380ee497e

BUG=chromium-os:8693
TEST=Ran unittests (cbuildbot, cros_mark_*) and ran cbuildbot with new chrome pfq items.

Review URL: http://codereview.chromium.org/5783001
2010-12-13 11:27:26 -08:00
Chris Sosa
c663f209f7 Add ability to store the manifest in the manifest directory.
Change-Id: I1ac3ed12df00424ba5ba2d1c61fd6fe9716e13b9

BUG=chromium-os:9714
TEST=Ran new unittests as well as ran cbuildbot on local b/cbuild

Review URL: http://codereview.chromium.org/5689003
2010-12-13 11:11:35 -08:00
David James
ab5c6917c6 Update cbuildbot.py to use emptytree option instead of deleting board.
This allows us to avoid deleting the board and still ensure our packages are
in sync with the full buildbot. This allows us to detect conflicts between
old and new packages.

BUG=chromium-os:10127
TEST=Run cbuildbot.py with emptytree feature forced to on and verify that
     build_packages rebuilds all packages. Also run unit tests.

Change-Id: I83e186937e28eef44285724697895a4583910b58

Review URL: http://codereview.chromium.org/5652007
2010-12-09 14:52:46 -08:00
Chris Sosa
1a4a9665d4 Have upload_prebuilts respect the debug flag.
Chrome PFQ right now is uploading prebuilts which wasn't meant to happen :(

Change-Id: I502a0621a6c9f655e5a3f890dd79e00652925902

BUG=chromium-os:8693
TEST=Ran cbuildbot with --debug

Review URL: http://codereview.chromium.org/5595004
2010-12-03 16:28:33 -08:00
Chris Sosa
55430beaef Add ability to push to subset of overlays we rev.
We also fix some of the configs for existing configurations.

Change-Id: I34baa9664ff9e91268c8bac976ab2ab0aee51bfb

BUG=chromium-os:9714
TEST=Running now with cbuildbot x86-generic-pre-flight-queue.

Review URL: http://codereview.chromium.org/5531002
2010-12-02 15:49:00 -08:00
David James
59a6600edd Update scripts to avoid using deprecated --test_case syntax.
TEST=Used cbuildbot.py to run tests
BUG=chromium-os:9893

Change-Id: I3c3856fee6fa09c4e2c829f3c8f23d098b4b6bb5

Review URL: http://codereview.chromium.org/5555003
2010-12-02 13:54:48 -08:00
David James
63c51c4d3c Update cbuildbot.py and prebuilt.py to deduplicate preflight prebuilts.
Preflight prebuilts reference the last full build, so they should always be complete as long as the last full build is still there.

Also add host prebuilts to the preflight prebuilts.

BUG=chromium-os:5311
TEST=Run unit tests for cbuildbot and prebuilt.py. Test runs of cbuildbot.py with --dryrun.

Review URL: http://codereview.chromium.org/5344002

Change-Id: Id95f94c02cc2f6cbd70a029d4f8b94617b7cc071
2010-11-30 11:45:11 -08:00
David James
c509a906da Update cbuildbot.py to upload prebuilts from preflight buildbot.
This change updates cbuildbot.py to upload prebuilts and reference them via PREFLIGHT_BINHOST.

Also ensure make.conf ends in a newline, as it looks nicer.

BUG=chromium-os:5311
TEST=Test that prebuilt.py updates PREFLIGHT_BINHOST in right make.conf
     file. Test that cbuildbot.py runs prebuilt.py with right
     arguments. Run unit tests for cbuildbot and prebuilt.py. Test runs of cbuildbot.py with --dryrun. Test upload of Packages file to local web server and check that emerge works with them.

Change-Id: Iad03c6c469e05b9ee1cceff69cbe2bdd51225e25

Review URL: http://codereview.chromium.org/4969003
2010-11-30 11:36:57 -08:00
Chris Sosa
564373521e Reconfigure to use ssh connection rather than use different manifest.
This also removes the rw_checkout var as it's not longer changes the affect of whether we use the ssh url or not.

Note that this replaces the other CL from last night because the other CL didn't work.  The other CL still pulled in the same manifest that has http: url's for each git repo.  I reverted that change last night after I noticed this.

Change-Id: I3d4ad2be6887ac2cf4ed2009bad9cae6dfdf5bbf

BUG=chromium-os:9509
TEST=Ran with cbuildbot --clobber and incremental

Review URL: http://codereview.chromium.org/5278010
2010-11-29 10:42:49 -08:00
Chris Sosa
c58667b8c4 This CL adds the ability to Rev and Build Chrome using cros_mark_chrome_as_stable.
Other things:
Removing commit|clean|push command from cros_mark_chrome because it's unnecessary ... I'm just gonna use the same stabilizing branch as the other stable marker, so I can use the same clean / push from the other script.  This makes this change fit in much more nicely to cbuildbot.

Other changes:
Add ability for cros_mark_chrome_as_stable to communicate to calling script through stdout.
Make STABLE_BRANCH in cros_mark_as_stable public for above reason.
Removed push_options from cros_mark and accompanying change to cbuildbot as it isn't used anymore.
Made it easier to debug with cbuildbot (I ran cbuildbot A LOT of times to test this change so I felt the pain)...this includes:
  - only build when syncing
  - allow one to explicitly disable tests
  - use dryrun is options.debug is set

Change-Id: I413a2e81a99cdde2e4d9139561cd518245b9b346

BUG=chromium-os:8693
TEST=Ran cbuildbot with --notest --debug --nosync x86-pre-flight and
saw it go through correctly and dryrun push.  Ran cbuildbot with same and --chrome_rev=tot.  Ran with syncing and running tests too.

Review URL: http://codereview.chromium.org/5154008
2010-11-24 13:18:39 -08:00
Thieu Le
ac13647eda Change _ArchiveTestResults to upload to Google Storage
BUG=chromium-os:8364
TEST=

Change-Id: Icecc8268fa2c12c8413caa2879785f00e4be5a0a

Review URL: http://codereview.chromium.org/4864001
2010-11-24 10:42:32 -08:00
Chris Sosa
444e3a6de0 Revert "Change url of manifest to use ssh one as opposed to mirrored repo."
This reverts commit 4847031faa.

TBR=Required for pushes to work.

Change-Id: I4f07407e518045ea77fa61942af58f57e68b5f8f
2010-11-24 00:54:48 -08:00
Chris Sosa
4847031faa Change url of manifest to use ssh one as opposed to mirrored repo.
Change-Id: I680e28bcafbfeb4f19e2404e2b28761e26111f7f

BUG=chromium-os:9509
TEST=Ran cbuildbot on my local builder checkout with both clobber and without.
It sync'd and built.

Review URL: http://codereview.chromium.org/5312002
2010-11-23 16:29:01 -08:00
Chris Sosa
e4836188a8 Add ability for cbuildbot to only run unit tests for revved packages.
Change-Id: I6744aac09fb075f91f710f16343fab45b8dc8af9

BUG=chromium-os:7283
TEST=Ran it with cbuildbot through the unit tests phase and looked
at the file and saw it get parsed correctly.

Review URL: http://codereview.chromium.org/5124006
2010-11-22 13:44:51 -08:00
David James
77f281a75e Move sanity checks for missing directories until after checkout happens.
Preflight clean needs to wait until after the checkout happens before
it asserts that all of our overlays exist.

BUG=chromium-os:9197
TEST=Ran cbuildbot.py with missing chromiumos-overlay dir

Change-Id: If66574f01d1e85741e971919a9fa2da34f85872d

Review URL: http://codereview.chromium.org/5069001
2010-11-15 20:39:00 -08:00
David James
b49c6df754 Add more error checking to preflight queue.
What's new?
  - cros_mark_as_stable now exits with errors if directories are specified
    that don't exist.
  - cbuildbot.py always explicitly specifies overlay directories so
    cros_mark_as_stable can rely on them existing.
  - Package names and paths are now separated with colons instead of
    spaces, so as to allow for us using the same syntax with
    enter_chroot.sh as we use without the same script. (enter_chroot.sh
    mucks with command-lines that contain spaces or quotes.)
  - cbuildbot.py now ensures its build path is a absolute path. This ensures we don't kill the wrong processes, if, for instance, the buildpath is '../..'
  - All buildbots now explicitly specify what overlays they want to rev. Public buildbots only rev public ebuilds and private buildbots now only rev private ebuilds.

BUG=chromium-os:8647
TEST=Ran unit tests. Manually marked packages as stable. Ran cbuildbot.py test run.

Change-Id: I1df6d428973d91329c4f5159e2886889a3ebb7c7

Committed: http://chrome-svn/viewvc/chromeos?view=rev&revision=795bd30

Review URL: http://codereview.chromium.org/4442001
2010-11-15 13:06:42 -08:00
David James
02b3e4f3d5 Revert "Add more error checking to preflight queue."
This reverts commit 795bd30f06.

BUG=
TEST=

Review URL: http://codereview.chromium.org/4904003

Change-Id: I9148026ecd0df75c253cfcf0c67af3d26771dd21
2010-11-12 14:47:57 -08:00
David James
795bd30f06 Add more error checking to preflight queue.
What's new?
  - cros_mark_as_stable now exits with errors if directories are specified
    that don't exist.
  - cbuildbot.py always explicitly specifies overlay directories so
    cros_mark_as_stable can rely on them existing.
  - Package names and paths are now separated with colons instead of
    spaces, so as to allow for us using the same syntax with
    enter_chroot.sh as we use without the same script. (enter_chroot.sh
    mucks with command-lines that contain spaces or quotes.)
  - cbuildbot.py now ensures its build path is a absolute path. This ensures we don't kill the wrong processes, if, for instance, the buildpath is '../..'
  - All buildbots now explicitly specify what overlays they want to rev. Public buildbots only rev public ebuilds and private buildbots now only rev private ebuilds.

BUG=chromium-os:8647
TEST=Ran unit tests. Manually marked packages as stable. Ran cbuildbot.py test run.

Change-Id: I1df6d428973d91329c4f5159e2886889a3ebb7c7

Review URL: http://codereview.chromium.org/4442001
2010-11-12 14:33:14 -08:00
David James
4beb5cb58f Revert "Add more error checking to preflight queue."
This reverts commit b4258c8bd4.

BUG=br0ken build
TBR=rspangler
TEST=

Review URL: http://codereview.chromium.org/4697007

Change-Id: I839ffd08d33fc5943696cfe31f2978fe32ce5fec
2010-11-11 10:51:07 -08:00
David James
f9b84d61a1 Define buildroot properly.
Followup to "Add more error checking to preflight queue."
BUG=Br0ken build
TBR=kliegs@chromium.org
TEST=Ran unit tests

Review URL: http://codereview.chromium.org/4742004

Change-Id: I63e52265ae404f0193cb1f9ab35f3f1ec266d2c1
2010-11-11 10:45:12 -08:00
David James
b4258c8bd4 Add more error checking to preflight queue.
What's new?
  - cros_mark_as_stable now exits with errors if directories are specified
    that don't exist.
  - cbuildbot.py always explicitly specifies overlay directories so
    cros_mark_as_stable can rely on them existing.
  - Package names and paths are now separated with colons instead of
    spaces, so as to allow for us using the same syntax with
    enter_chroot.sh as we use without the same script. (enter_chroot.sh
    mucks with command-lines that contain spaces or quotes.)
  - All buildbots now explicitly specify what overlays they want to rev. Public buildbots only rev public ebuilds and private buildbots now only rev private ebuilds.

BUG=chromium-os:8647
TEST=Ran unit tests. Manually marked packages as stable.

Change-Id: I1df6d428973d91329c4f5159e2886889a3ebb7c7

Review URL: http://codereview.chromium.org/4442001
2010-11-11 10:37:04 -08:00
David James
1f79aafb89 Fix broken quoting in cbuildbot.py, allowing for correct revving of ebuilds.
When you pass '--foo="bar baz"' to a script, the argument is literally
passed as "bar baz", with the quotes. So we need to not add unnecessary
quotes in there.

When enter_chroot.sh parses arguments, it actually removes the quotes
for you, and splits up arguments. As a result, it may actually
be necessary to (1) use the above broken syntax when you call scripts
using enter_chroot.sh; but (2) don't use that syntax when calling
scripts without enter_chroot.sh. That's a broken situation, so I've
added a TODO for that. I've also added more warnings to
cros_mark_as_stable.py so that it's easier to debug when it doesn't
work.

BUG=chromium-os:8633
TEST=Ran unit tests. Planning to watch TOT buildbot post-commit to make sure behavior is right.

Change-Id: Ia1a0e60153fec60cb7ed4db2da128ffd9ae81e23

Review URL: http://codereview.chromium.org/4385002
2010-11-03 11:15:11 -07:00
David James
3d57ef8139 Update cros_mark_as_stable to support working on only private overlays.
BUG=chromium-os:8493
TEST=Ran unit tests. Verified that cros_mark_as_stable.py supports
running on just the private or public overlay.

Change-Id: I5e2e8fc5e63b3c93bb688a5ce87233315c0c2d1c

Review URL: http://codereview.chromium.org/4250001
2010-11-01 16:22:35 -07:00
David James
d2b72f7c2f Correctly initialize tracking branch in preflight rinse.
Currently cros/master is used for the tracking branch in the preflight
rinse step. This is wrong if --tracking_branch is specified
differently in the command-line options to cbuildbot.py.

BUG=none
TEST=Ran unit tests

Change-Id: Id4fb989863ae11f57889faf6a8d8b043b81da58e

Review URL: http://codereview.chromium.org/4104005
2010-10-26 12:35:37 -07:00
Chris Sosa
b00c32c335 Fix the tree by stating to right path.
Change-Id: I33e861066e8a585ce839ebc7a8c23243fb18cd1c

BUG=8083
TEST=Ran unit tests

Review URL: http://codereview.chromium.org/4053007
2010-10-23 23:38:48 -07:00
Chris Sosa
4a6e482b31 Gzip using the fastest option.
Change-Id: Id75e54879e896e42ef69659fc602f625858fd33f

BUG=
TEST=

Review URL: http://codereview.chromium.org/3989004
2010-10-22 17:30:21 -07:00
Chris Sosa
7e493cb4d3 Archive test results and image after running vm suite.
Change-Id: Ic6d4a79a7ab58b8043805a6b826f251faa9f3a6c

BUG=
TEST=Running now (adding unit tests as well)

Review URL: http://codereview.chromium.org/4062003
2010-10-22 16:32:30 -07:00
Chris Sosa
6b6f35ec65 Always pass board to cros_mark_as_stable.
Change-Id: I9a724e4e20b8945fdaa1dc53afdddf2477a5bbab

BUG=
TEST=Ran unit tests.

Review URL: http://codereview.chromium.org/3956007
2010-10-22 16:14:36 -07:00
Anush Elangovan
a6b1390fc5 Add branch support to preflight
Change-Id: Ia2592c6384e94e4e8e5400eddbf2e03aac150eef

BUG=7905
TEST=run cbuildbot on my workstation.

Review URL: http://codereview.chromium.org/4040002
2010-10-22 09:24:43 -07:00
Chris Sosa
dcd1de0bb9 Change preflight cleanup process to always be run before any run and add lazy unmount before every run.
Change-Id: If7db175f828fccdd4caa342a0370e842503f16dc

BUG=8015
TEST=Ran it with cbuildbot with/without --clobber with/without old state.

Review URL: http://codereview.chromium.org/4007005
2010-10-21 18:22:50 -07:00
David James
6be0f08169 Robustify and speed up cros_mark_all_as_stable.
What's new?
 - cros_mark_all_as_stable is now in Python and has unit tests.
 - We now detect and report coding errors that can cause incorrect behavior.
   E.g., if 9999 ebuild or stable ebuild can't be found, we report a hard
   failure so that developers will know about the problem.
 - We now check that git hashes we report actually match up with the
   right repository.
 - Unified diff of changes are now printed to stdout, so that developers
   can see a list of the changes and debug any problems.
 - cros_mark_all_as_stable now takes 2.5 seconds to run.

BUG=chromium-os:7795
TEST=Manually examined diff output from run of cros_mark_all_as_stable
     on full repository.

Change-Id: I762597c9b94e5f8e8171b83c966ad54e21a65c1b

Review URL: http://codereview.chromium.org/3798003
2010-10-21 16:11:00 -07:00
Chris Sosa
aa4670315b Fix equals in RunCommand's in cbuildbot.
Change-Id: If8cdde42a470c3d07cfd41552cf95dea7567688b

BUG=
TEST=Ran cbuildbot.

Review URL: http://codereview.chromium.org/3967002
2010-10-21 10:54:02 -07:00
Scott Zawalski
9310bb6bd3 Fix logic for debug and uprev
BUG=chromium-os:8000
TEST=NA

Review URL: http://codereview.chromium.org/4007003
2010-10-20 22:36:37 -07:00
Chris Sosa
debca51332 Add --debug option to cbuildbot for developers to use cbuildbot without pushing changes.
Change-Id: Idda6c7287cdb1863eb4abe8c56da2e763e9fe777

BUG=7926
TEST=Ran with --debug and verified I didn't push a change.

Review URL: http://codereview.chromium.org/3880002
2010-10-20 11:00:44 -07:00
Chris Sosa
5911f2003b Clean up any previous kvm state.
Change-Id: I01fecde5e687992ccf36889803a4cb84b71a38b2

BUG=7818
TEST=Ran with/without previous kvm instance

Review URL: http://codereview.chromium.org/3760012
2010-10-19 12:58:24 -07:00
David James
ef9f04e2fd Update cbuildbot to use git.chromium.org instead of src.chromium.org.
src.chromium.org is no longer supported for git operations.

BUG=chromium-os:7830
TEST=none

Change-Id: Ia7cd9ff76625441f16d20d2f9a8e0f752dca0586

Review URL: http://codereview.chromium.org/3782010
2010-10-15 16:18:29 -07:00
Anush Elangovan
189d85d20e Revert "Fix issue with stale cache from bad builds."
This reverts commit 365d4b0cc8.

Fix preflight breakage, this seems like the most possible cause. Will recommit if not.

BUG=
TEST=

Review URL: http://codereview.chromium.org/3767012

Change-Id: I93fe463fec77c445e7ff0ec86db25d5ef2997ad7
2010-10-15 11:36:53 -07:00