Commit Graph

49 Commits

Author SHA1 Message Date
David James
795bd30f06 Add more error checking to preflight queue.
What's new?
  - cros_mark_as_stable now exits with errors if directories are specified
    that don't exist.
  - cbuildbot.py always explicitly specifies overlay directories so
    cros_mark_as_stable can rely on them existing.
  - Package names and paths are now separated with colons instead of
    spaces, so as to allow for us using the same syntax with
    enter_chroot.sh as we use without the same script. (enter_chroot.sh
    mucks with command-lines that contain spaces or quotes.)
  - cbuildbot.py now ensures its build path is a absolute path. This ensures we don't kill the wrong processes, if, for instance, the buildpath is '../..'
  - All buildbots now explicitly specify what overlays they want to rev. Public buildbots only rev public ebuilds and private buildbots now only rev private ebuilds.

BUG=chromium-os:8647
TEST=Ran unit tests. Manually marked packages as stable. Ran cbuildbot.py test run.

Change-Id: I1df6d428973d91329c4f5159e2886889a3ebb7c7

Review URL: http://codereview.chromium.org/4442001
2010-11-12 14:33:14 -08:00
David James
4beb5cb58f Revert "Add more error checking to preflight queue."
This reverts commit b4258c8bd4.

BUG=br0ken build
TBR=rspangler
TEST=

Review URL: http://codereview.chromium.org/4697007

Change-Id: I839ffd08d33fc5943696cfe31f2978fe32ce5fec
2010-11-11 10:51:07 -08:00
David James
f9b84d61a1 Define buildroot properly.
Followup to "Add more error checking to preflight queue."
BUG=Br0ken build
TBR=kliegs@chromium.org
TEST=Ran unit tests

Review URL: http://codereview.chromium.org/4742004

Change-Id: I63e52265ae404f0193cb1f9ab35f3f1ec266d2c1
2010-11-11 10:45:12 -08:00
David James
b4258c8bd4 Add more error checking to preflight queue.
What's new?
  - cros_mark_as_stable now exits with errors if directories are specified
    that don't exist.
  - cbuildbot.py always explicitly specifies overlay directories so
    cros_mark_as_stable can rely on them existing.
  - Package names and paths are now separated with colons instead of
    spaces, so as to allow for us using the same syntax with
    enter_chroot.sh as we use without the same script. (enter_chroot.sh
    mucks with command-lines that contain spaces or quotes.)
  - All buildbots now explicitly specify what overlays they want to rev. Public buildbots only rev public ebuilds and private buildbots now only rev private ebuilds.

BUG=chromium-os:8647
TEST=Ran unit tests. Manually marked packages as stable.

Change-Id: I1df6d428973d91329c4f5159e2886889a3ebb7c7

Review URL: http://codereview.chromium.org/4442001
2010-11-11 10:37:04 -08:00
David James
1f79aafb89 Fix broken quoting in cbuildbot.py, allowing for correct revving of ebuilds.
When you pass '--foo="bar baz"' to a script, the argument is literally
passed as "bar baz", with the quotes. So we need to not add unnecessary
quotes in there.

When enter_chroot.sh parses arguments, it actually removes the quotes
for you, and splits up arguments. As a result, it may actually
be necessary to (1) use the above broken syntax when you call scripts
using enter_chroot.sh; but (2) don't use that syntax when calling
scripts without enter_chroot.sh. That's a broken situation, so I've
added a TODO for that. I've also added more warnings to
cros_mark_as_stable.py so that it's easier to debug when it doesn't
work.

BUG=chromium-os:8633
TEST=Ran unit tests. Planning to watch TOT buildbot post-commit to make sure behavior is right.

Change-Id: Ia1a0e60153fec60cb7ed4db2da128ffd9ae81e23

Review URL: http://codereview.chromium.org/4385002
2010-11-03 11:15:11 -07:00
David James
3d57ef8139 Update cros_mark_as_stable to support working on only private overlays.
BUG=chromium-os:8493
TEST=Ran unit tests. Verified that cros_mark_as_stable.py supports
running on just the private or public overlay.

Change-Id: I5e2e8fc5e63b3c93bb688a5ce87233315c0c2d1c

Review URL: http://codereview.chromium.org/4250001
2010-11-01 16:22:35 -07:00
David James
d2b72f7c2f Correctly initialize tracking branch in preflight rinse.
Currently cros/master is used for the tracking branch in the preflight
rinse step. This is wrong if --tracking_branch is specified
differently in the command-line options to cbuildbot.py.

BUG=none
TEST=Ran unit tests

Change-Id: Id4fb989863ae11f57889faf6a8d8b043b81da58e

Review URL: http://codereview.chromium.org/4104005
2010-10-26 12:35:37 -07:00
Chris Sosa
b00c32c335 Fix the tree by stating to right path.
Change-Id: I33e861066e8a585ce839ebc7a8c23243fb18cd1c

BUG=8083
TEST=Ran unit tests

Review URL: http://codereview.chromium.org/4053007
2010-10-23 23:38:48 -07:00
Chris Sosa
4a6e482b31 Gzip using the fastest option.
Change-Id: Id75e54879e896e42ef69659fc602f625858fd33f

BUG=
TEST=

Review URL: http://codereview.chromium.org/3989004
2010-10-22 17:30:21 -07:00
Chris Sosa
7e493cb4d3 Archive test results and image after running vm suite.
Change-Id: Ic6d4a79a7ab58b8043805a6b826f251faa9f3a6c

BUG=
TEST=Running now (adding unit tests as well)

Review URL: http://codereview.chromium.org/4062003
2010-10-22 16:32:30 -07:00
Chris Sosa
6b6f35ec65 Always pass board to cros_mark_as_stable.
Change-Id: I9a724e4e20b8945fdaa1dc53afdddf2477a5bbab

BUG=
TEST=Ran unit tests.

Review URL: http://codereview.chromium.org/3956007
2010-10-22 16:14:36 -07:00
Anush Elangovan
a6b1390fc5 Add branch support to preflight
Change-Id: Ia2592c6384e94e4e8e5400eddbf2e03aac150eef

BUG=7905
TEST=run cbuildbot on my workstation.

Review URL: http://codereview.chromium.org/4040002
2010-10-22 09:24:43 -07:00
Chris Sosa
dcd1de0bb9 Change preflight cleanup process to always be run before any run and add lazy unmount before every run.
Change-Id: If7db175f828fccdd4caa342a0370e842503f16dc

BUG=8015
TEST=Ran it with cbuildbot with/without --clobber with/without old state.

Review URL: http://codereview.chromium.org/4007005
2010-10-21 18:22:50 -07:00
David James
6be0f08169 Robustify and speed up cros_mark_all_as_stable.
What's new?
 - cros_mark_all_as_stable is now in Python and has unit tests.
 - We now detect and report coding errors that can cause incorrect behavior.
   E.g., if 9999 ebuild or stable ebuild can't be found, we report a hard
   failure so that developers will know about the problem.
 - We now check that git hashes we report actually match up with the
   right repository.
 - Unified diff of changes are now printed to stdout, so that developers
   can see a list of the changes and debug any problems.
 - cros_mark_all_as_stable now takes 2.5 seconds to run.

BUG=chromium-os:7795
TEST=Manually examined diff output from run of cros_mark_all_as_stable
     on full repository.

Change-Id: I762597c9b94e5f8e8171b83c966ad54e21a65c1b

Review URL: http://codereview.chromium.org/3798003
2010-10-21 16:11:00 -07:00
Chris Sosa
aa4670315b Fix equals in RunCommand's in cbuildbot.
Change-Id: If8cdde42a470c3d07cfd41552cf95dea7567688b

BUG=
TEST=Ran cbuildbot.

Review URL: http://codereview.chromium.org/3967002
2010-10-21 10:54:02 -07:00
Scott Zawalski
9310bb6bd3 Fix logic for debug and uprev
BUG=chromium-os:8000
TEST=NA

Review URL: http://codereview.chromium.org/4007003
2010-10-20 22:36:37 -07:00
Chris Sosa
debca51332 Add --debug option to cbuildbot for developers to use cbuildbot without pushing changes.
Change-Id: Idda6c7287cdb1863eb4abe8c56da2e763e9fe777

BUG=7926
TEST=Ran with --debug and verified I didn't push a change.

Review URL: http://codereview.chromium.org/3880002
2010-10-20 11:00:44 -07:00
Chris Sosa
5911f2003b Clean up any previous kvm state.
Change-Id: I01fecde5e687992ccf36889803a4cb84b71a38b2

BUG=7818
TEST=Ran with/without previous kvm instance

Review URL: http://codereview.chromium.org/3760012
2010-10-19 12:58:24 -07:00
David James
ef9f04e2fd Update cbuildbot to use git.chromium.org instead of src.chromium.org.
src.chromium.org is no longer supported for git operations.

BUG=chromium-os:7830
TEST=none

Change-Id: Ia7cd9ff76625441f16d20d2f9a8e0f752dca0586

Review URL: http://codereview.chromium.org/3782010
2010-10-15 16:18:29 -07:00
Anush Elangovan
189d85d20e Revert "Fix issue with stale cache from bad builds."
This reverts commit 365d4b0cc8.

Fix preflight breakage, this seems like the most possible cause. Will recommit if not.

BUG=
TEST=

Review URL: http://codereview.chromium.org/3767012

Change-Id: I93fe463fec77c445e7ff0ec86db25d5ef2997ad7
2010-10-15 11:36:53 -07:00
Chris Sosa
365d4b0cc8 Fix issue with stale cache from bad builds.
We always want to re-build packages we are revving, however, if the last
build succeeds in building but fails tests, we have a bad local cache that
has revved packages.  This avoids this issue by working on all packages we
are revving.

Change-Id: I3bd7463a4090b1f007d09be81bdf65657bd9c3f3

BUG=7589
TEST=Ran it with cbuildbot.

Review URL: http://codereview.chromium.org/3745006
2010-10-15 10:37:04 -07:00
Chris Sosa
4befe39060 Revert "Revert "Close tree when smoke bvt fails.""
This reverts commit 224f7548b5.

BUG=
TEST=

Review URL: http://codereview.chromium.org/3525022

Change-Id: Ib4e136b96a1d9dac7345598b281b053a47421e77
2010-10-07 19:04:05 -07:00
Chris Sosa
3ede50ed4e Add enabling of local account. This should get the suite_Smoke to go green on the pfq.
Change-Id: Id480f45836a81d7d994b92d7a45d16208e1dd11c

BUG=
TEST=Ran locally.  Saw all tests pass.

Review URL: http://codereview.chromium.org/3567010
2010-10-01 14:12:26 -07:00
Chris Sosa
c2db26c796 Add methods to run vm tests on x86-pre-flight-queue.
Right now we're leaving error_ok=True until the tests are shown to work on builder.

Change-Id: I6b8c690a0da30948389fd4312032c78d87115364

BUG=6906
TEST=Ran through it locally.

Review URL: http://codereview.chromium.org/3591002
2010-09-30 14:25:25 -07:00
David James
70b297a263 Update cbuildbot.py to print manifests when we sync.
If cbuildbot.py printed out a manifest when it syncs, it would make it possible
for us to reproduce the build later by looking at what revisions the buildbot
was using. This is useful for debugging.

Change-Id: I186ad01eef0090b6c618cb7ad583085197474018

BUG=chromium-os:7069
TEST=Ran test suite with 3441027 patched in (Sosa's patch that fixes an unrelated issue with the unit tests)

Review URL: http://codereview.chromium.org/3461028
2010-09-25 23:25:18 -07:00
David James
7bf8d2715e Switch off repo sync --jobs, and turn on repo --trace.
repo sync --jobs doesn't check for errors, so we shouldn't use it in
automated scripts. repo --trace is needed so that we can diagnose
whether git is hanging.

BUG=chromium-os:7048, chromium-os:6774
TEST=Ran test suite

Change-Id: I9e68cfffe841a231f5fabef951ea2d45b81c1d5e

Review URL: http://codereview.chromium.org/3419024
2010-09-25 23:23:00 -07:00
Chris Sosa
7de6531962 Faster checkout and cleanup of old builds.
Change-Id: Id81fe4e7bd9cae7a51a45b6ea8a535c4ea9430f8

BUG=
TEST=Ran locally on top of previous build area.

Review URL: http://codereview.chromium.org/3337007
2010-09-02 15:10:48 -07:00
Chris Sosa
e13960240b Fix cleanup so we don't have to clobber as much when failures occur
Change-Id: I120c49a38d2a17a2216296a68378fee7345f616a

TBR=petkov

Review URL: http://codereview.chromium.org/3255004
2010-08-30 22:37:28 -07:00
Chris Sosa
aaba25d817 Fix clobber. Strip any extra args and treat last arg as config.
Change-Id: I2bf951342c12d954bd1abbc8b878437c648b4ac7

TBR=Scottz
2010-08-30 20:28:11 -07:00
Chris Sosa
70e2c9e72a Move RunCommand, and Info/Warning/Die into common pylib
TEST=Ran all commands that used RunCommand or Die with
their associated unittests.

Review URL: http://codereview.chromium.org/3266004
2010-08-30 11:05:50 -07:00
Chris Sosa
1ec70345d9 Change clobber logic to be explicit
TEST=Tested with --clobber and without.

Change-Id: I6a938b3da673d97186cec79f25b22ea6176f1fdf

Review URL: http://codereview.chromium.org/3238012
2010-08-30 10:48:08 -07:00
Chris Sosa
0c13b46899 Move back to marking all until we have individual tree closing ready.
Review URL: http://codereview.chromium.org/3246003
2010-08-27 17:01:38 -07:00
Chris Sosa
0c156f9ccb TBR: Add replace to build_image 2010-08-27 13:11:56 -07:00
Chris Sosa
151a896d4b TBR: Fix missing enter_chroot's in pfq 2010-08-27 11:48:08 -07:00
Chris Sosa
6d58d427e3 Add all pfq configurations and add build_image / run_unit_tests.
BUG=5468
TEST=Ran default and ran x86-generic.  Took board values from master.cfg in buildbot

Review URL: http://codereview.chromium.org/3200006

Change-Id: I1d3e4216dc9ca9af0d9e8c1bfffabffa87f89b04
2010-08-27 10:04:31 -07:00
Chris Sosa
4b9baf9fa1 Add ability for cbuildbot master to synchronize with cbuildbot slaves.
TEST=Ran several non-checked in unit tests (will check in later) and tested
with cbuildbot as it is now.
BUG=5989

Review URL: http://codereview.chromium.org/3165052

Change-Id: I867a597b34bbf63261b9d19b57cef9631100c339
2010-08-26 14:06:56 -07:00
Chris Sosa
6f7d6cef0e Fix case where there is nothing to uprev.
TEST=Ran unit tests.

Change-Id: Iafd9702c5364511d79667ecc296eaf68fc89a5fb

Review URL: http://codereview.chromium.org/3159047
2010-08-26 12:34:20 -07:00
Chris Sosa
5140c15674 Fix bug with git packages with no matching ebuild.
Change-Id: I45454c837807fbe6e5996b76087bbfe25d7155ce

Review URL: http://codereview.chromium.org/3171032
2010-08-26 11:18:25 -07:00
Chris Sosa
e9b94eefd0 CBuildbot - Adds ability to only rev packages given by the sourcestamp.
The buildbot drops a source stamp (repo1.git@commit1 repo2.git@commit2 ...).
If CBuildbot finds the source stamp, it uses it and revs only those packages.
If this file is either missing or set to "None", it assumes a Force Build
has been pressed and keeps the old behavior (attempts to mark all).

TEST=Ran with 3 states of revisions file.
BUG=5006

Review URL: http://codereview.chromium.org/3163030

Change-Id: I226fd3bec642224b31ce51eee34d028043964943
2010-08-26 10:55:27 -07:00
Mandeep Singh Baines
a87699b198 cbuildbot: fix to point to new url
BUG=6102
TEST=none

Change-Id: I69312fed2d0eb4934a964792b4af06160b8d8eb5

Review URL: http://codereview.chromium.org/3135049
2010-08-25 14:39:18 -07:00
Chris Sosa
a8a4803f4f Fix cbuildbot. Mistakenly used str.open instead of open
TEST=Test underway.

Review URL: http://codereview.chromium.org/3116029
2010-08-20 10:55:48 -07:00
Mandeep Singh Baines
2e9e668dc0 cbuildbot: fix typo
BUG=5937
TEST=none

Change-Id: I5b4a79fff4f3d263812e1dd0392e21ec3bede69d

Review URL: http://codereview.chromium.org/3175030
2010-08-20 08:08:43 -07:00
Chris Sosa
81dae09de0 Add a flag to pick up timestamp of commits and print information if found and add noclobber option for easier testing.
TEST=Ran cbuildbot with file.

Review URL: http://codereview.chromium.org/3174021
2010-08-19 18:03:06 -07:00
Chris Sosa
d26c87d176 Buildbots should not need to push changes unless they are a pre flight buildbot.
TEST=Ran cbuildbot twice.
BUG=5907

Review URL: http://codereview.chromium.org/3110027
2010-08-19 16:20:51 -07:00
Chris Sosa
1ee50d3ce2 Add repo sync all helper for non-pfq buildbots.
TEST=Ran script and re-ran cbuildbot to completion of sync.

Change-Id: I31fddca6f720fecb461a96c1ffa54d4923b3c1c4

Review URL: http://codereview.chromium.org/3166019
2010-08-18 21:25:05 -07:00
Chris Sosa
9a27b4099f Always re-configure to use ssh even on incremental checkouts
Change-Id: I0a729b14e34c61a8174edd63fb20066e0811128e

Review URL: http://codereview.chromium.org/3181025
2010-08-18 18:18:18 -07:00
Chris Sosa
82047a9d48 Add some information to cbuildbot output.
TEST=Ran cbuildbot locally.

Review URL: http://codereview.chromium.org/3137006

Change-Id: I8c80334480a4a2e4bb32234d323c0b129bed4692
2010-08-12 10:07:35 -07:00
Mandeep Singh Baines
6718164717 cbuildbot: use correct port
Ugh, it worked on my machine because of my .ssh/config but need to
explicitly specify port for it to work on buildbot.

TBR=sosa
BUG=5532
TEST=This change was in the previous location of cbuilbot.py and working
correctly.

Change-Id: Id53a76bcb5d43c37ac725267864048d833f5a6ee

Review URL: http://codereview.chromium.org/3144005
2010-08-11 10:29:56 -07:00
Chris Sosa
cd358fe1cb Adding cbuildbot to cros_utils. Leaving old version in repo until buildbot changes over.
TEST=Ran locally outside of chroot.  Required change to remove Exception OSError as
e because "as" is a Python 2.6'ism

Review URL: http://codereview.chromium.org/3146001
2010-08-09 19:33:22 -07:00