diff --git a/contrib/jenkins/README.md b/contrib/jenkins/README.md new file mode 100644 index 0000000000..04d87c077a --- /dev/null +++ b/contrib/jenkins/README.md @@ -0,0 +1,72 @@ +# CoreOS Jenkins build system + +## Requirements + +### Storage Requirements + +CoreOS manifests are managed using git and repo, so a git server must be available to host these. An additional git repo is used by the `os-manifest` job to store a temporary manifest commit that is passed through to downstream jobs. + +The jobs also require google storage buckets for storing build artifacts such as binary packages and CoreOS images. + +### Jenkins Requirements + +The jobs use a number of Jenkins plugins during jobs. These are: + +- [Git](https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin) +- [Rebuilder](https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin) +- [Parameterized Trigger](https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin) +- [Copy Artifact](https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin) +- [SSH Agent](https://wiki.jenkins-ci.org/display/JENKINS/SSH+Agent+Plugin) +- [Job Restrictions](https://wiki.jenkins-ci.org/display/JENKINS/Job+Restrictions+Plugin) +- [Credentials Binding](https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin) +- [TAP](https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin) - for collecting test results from kola +- [Matrix Project](https://wiki.jenkins-ci.org/display/JENKINS/Matrix+Project+Plugin) + +### Slave Requirements + +The Jenkins jobs assume that each build slave is running CoreOS. The scripts that execute as part of the jobs use a number of tools present in CoreOS. Different host operating systems are untested. + +All jobs assume that the Jenkins user on the slaves have `sudo` access, which is used by the `cork` tool. + +The Jenkins slave used to execute the `os-kola-qemu` job must be run on a host that has KVM, so this slave cannot be in a VM unless it is using nested KVM, which is untested. + +Most jobs use the slave label `coreos` to execute on a CoreOS system, so at least one slave (or the master) is required with this label. The `os-kola-qemu` job requires a slave with the `coreos` *and* `kvm` label. + +### Secret Requirements + +Some secrets are required for the various things done during the build: + +- Slave SSH keys (if Jenkins slaves are used) +- git ssh key for manifest-build pushes from `os-manifest` +- google storage api key for artifact uploads and downloads + +## Setup + +### Running Jenkins + +If you have no Jenkins instance, you can run one in a [Docker container](https://hub.docker.com/_/jenkins/) on CoreOS: + +```sh +docker run -p 8080:8080 -p 50000:50000 jenkins +``` + +To create a slave, you will need to run CoreOS and either give Jenkins Oracle.com credentials to automatically install the JDK, or copy a JDK tarball to the CoreOS system and set `$JAVA_HOME` in the slave settings. + +### Install plugins and jobs + +Jenkins jobs in XML format are available in the [`jobs`](jobs) directory. A script called [`install.sh`](install.sh) is provided to copy jobs and install Jenkins plugins. The script will restart Jenkins as required by some plugins. + +For example, if the Jenkins instance is at `http://127.0.0.1:8080`: + +```sh +./install.sh http://127.0.0.1:8080 +``` + +### Configuring CoreOS jobs + +Some jobs will require modification to work correctly in any setup outside CoreOS. + +- `os-manifest` will need the git url for the `manifest-builds` repo. You will also need to configure the correct SSH secret for git pushes. +- Any job using google storage will need `GOOGLE_APPLICATION_CREDENTIALS` configured, which points to a [JSON Service Account key](https://cloud.google.com/storage/docs/authentication). Additionally, these jobs will need to point to your specific google storage buckets. +- Any job signing artifacts will need `GPG_SECRET_KEY_FILE`, which is a GPG private key used to sign built artifacts. + diff --git a/contrib/jenkins/install.sh b/contrib/jenkins/install.sh new file mode 100755 index 0000000000..e02c3acf8c --- /dev/null +++ b/contrib/jenkins/install.sh @@ -0,0 +1,24 @@ +#!/bin/sh -e + +if [ -z $1 ]; then + echo "Pass Jenkins URL" + exit 1 +fi + +JENKINS_URL=$1 + +for PLUGIN in git github rebuild parameterized-trigger copyartifact ssh-agent job-restrictions credentials-binding tap matrix-project; do + echo "Installing $PLUGIN..." + curl --silent --header "Content-Type: application/xml" -XPOST "$JENKINS_URL/pluginManager/installNecessaryPlugins" --data "" >/dev/null +done + +for JOB in $(find jobs/ -mindepth 1 -maxdepth 1 -type d); do + J=$(basename $JOB) + echo "Creating job $J..." + curl --silent --header "Content-Type: application/xml" -XPOST "$JENKINS_URL/createItem?name=$J" --data-binary "@$JOB/config.xml" >/dev/null +done + +curl -XPOST $JENKINS_URL/updateCenter/safeRestart + +echo "Visit $JENKINS_URL/updateCenter and wait for Jenkins to restart." + diff --git a/contrib/jenkins/jobs/os-board-image/config.xml b/contrib/jenkins/jobs/os-board-image/config.xml new file mode 100644 index 0000000000..2f3642cc23 --- /dev/null +++ b/contrib/jenkins/jobs/os-board-image/config.xml @@ -0,0 +1,179 @@ + + + + + + 30 + 50 + -1 + -1 + + false + + + + + BOARD + Target board to build + + + amd64-usr + arm64-usr + + + + + MANIFEST_URL + + https://github.com/coreos/manifest-builds.git + + + MANIFEST_REF + + refs/tags/ + + + MANIFEST_NAME + + release.xml + + + COREOS_OFFICIAL + + + + 0 + 1 + + + + + + + false + false + + + + coreos + false + false + false + false + + true + + + os-mantle + + + + + false + + + #!/bin/bash -ex + +# build may not be started without a ref value +[[ -n "${MANIFEST_REF#refs/tags/}" ]] + +# first thing, clear out old images +sudo rm -rf src/build + +script() { + local script="/mnt/host/source/src/scripts/${1}"; shift + ./bin/cork enter --experimental -- "${script}" "$@" +} + +./bin/cork update --create --downgrade-replace --verify --verbose \ + --manifest-url "${MANIFEST_URL}" \ + --manifest-branch "${MANIFEST_REF}" \ + --manifest-name "${MANIFEST_NAME}" + +source .repo/manifests/version.txt +export COREOS_BUILD_ID + +# Set up GPG for signing images +export GNUPGHOME="${PWD}/.gnupg" +rm -rf "${GNUPGHOME}" +trap "rm -rf '${GNUPGHOME}'" EXIT +mkdir --mode=0700 "${GNUPGHOME}" +gpg --import "${GPG_SECRET_KEY_FILE}" + +script setup_board --board=${BOARD} \ + --getbinpkgver="${COREOS_VERSION}" \ + --regen_configs_only + +if [[ "${COREOS_OFFICIAL}" -eq 1 ]]; then + GROUP=stable + UPLOAD=gs://builds.release.core-os.net/stable + script set_official --board=${BOARD} --official +else + GROUP=developer + UPLOAD=gs://builds.developer.core-os.net + script set_official --board=${BOARD} --noofficial +fi + +script build_image --board=${BOARD} \ + --group=${GROUP} \ + --getbinpkg \ + --getbinpkgver="${COREOS_VERSION}" \ + --sign=buildbot@coreos.com \ + --sign_digests=buildbot@coreos.com \ + --upload_root=${UPLOAD} \ + --upload prod container + +if [[ "${COREOS_OFFICIAL}" -eq 1 ]]; then + script image_set_group --board=${BOARD} \ + --group=alpha \ + --sign=buildbot@coreos.com \ + --sign_digests=buildbot@coreos.com \ + --upload_root=gs://builds.release.core-os.net/alpha \ + --upload + script image_set_group --board=${BOARD} \ + --group=beta \ + --sign=buildbot@coreos.com \ + --sign_digests=buildbot@coreos.com \ + --upload_root=gs://builds.release.core-os.net/beta \ + --upload +fi + + + + + false + + + chroot/build/${BOARD}/var/lib/portage/pkgs/*/*.tbz2,chroot/var/lib/portage/pkgs/*/*.tbz2,src/build/images/${BOARD}/latest/* + + + + + + + + BOARD=="${BOARD}" + + + os-board-vm-matrix,os-kola-qemu + UNSTABLE_OR_BETTER + false + + + + + + + + + 9de82aed-ae50-4233-989f-d454b0449b96 + GOOGLE_APPLICATION_CREDENTIALS + + + 12e071a3-694f-40e8-8595-489236d943cc + GPG_SECRET_KEY_FILE + + + + + diff --git a/contrib/jenkins/jobs/os-board-packages/config.xml b/contrib/jenkins/jobs/os-board-packages/config.xml new file mode 100644 index 0000000000..98c72718b5 --- /dev/null +++ b/contrib/jenkins/jobs/os-board-packages/config.xml @@ -0,0 +1,156 @@ + + + + Build binary packages for a target board. + + 30 + 50 + -1 + -1 + + false + + + + + BOARD + Target board to build + + + amd64-usr + arm64-usr + + + + + MANIFEST_URL + + https://github.com/coreos/manifest-builds.git + + + MANIFEST_REF + + refs/tags/ + + + MANIFEST_NAME + + release.xml + + + COREOS_OFFICIAL + + + + 0 + 1 + + + + + + + false + false + + + + coreos + false + false + false + false + + true + + + os-mantle + + + + + true + UseGlobalSetting + true + + false + + + #!/bin/bash -ex + +# build may not be started without a ref value +[[ -n "${MANIFEST_REF#refs/tags/}" ]] + +# use a ccache dir that persists across sdk recreations +# XXX: alternatively use a ccache dir that is usable by all jobs on a given node. +mkdir -p .cache/ccache + +enter() { + ./bin/cork enter --experimental -- env \ + CCACHE_DIR="/mnt/host/source/.cache/ccache" \ + CCACHE_MAXSIZE="5G" "$@" +} + +script() { + local script="/mnt/host/source/src/scripts/${1}"; shift + enter "${script}" "$@" +} + +./bin/cork update --create --downgrade-replace --verify --verbose \ + --manifest-url "${MANIFEST_URL}" \ + --manifest-branch "${MANIFEST_REF}" \ + --manifest-name "${MANIFEST_NAME}" + +source .repo/manifests/version.txt +export COREOS_BUILD_ID + +# figure out if ccache is doing us any good in this scheme +enter ccache --zero-stats + +script update_chroot --toolchain_boards=${BOARD} +if [[ "${COREOS_OFFICIAL:-0}" -eq 1 ]]; then + script setup_board --board=${BOARD} \ + --skip_chroot_upgrade \ + --toolchainpkgonly \ + --force +fi +script build_packages --board=${BOARD} \ + --skip_chroot_upgrade \ + --getbinpkgver=${COREOS_VERSION} \ + --toolchainpkgonly \ + --upload --upload_root gs://builds.developer.core-os.net + +enter ccache --show-stats + + + + + false + + + chroot/build/${BOARD}/var/lib/portage/pkgs/*/*.tbz2,chroot/var/lib/portage/pkgs/*/*.tbz2 + + + + + + + + os-board-image + UNSTABLE_OR_BETTER + false + + + + + + + + + 9de82aed-ae50-4233-989f-d454b0449b96 + GOOGLE_APPLICATION_CREDENTIALS + + + + + diff --git a/contrib/jenkins/jobs/os-board-vm-matrix/config.xml b/contrib/jenkins/jobs/os-board-vm-matrix/config.xml new file mode 100644 index 0000000000..8d50222502 --- /dev/null +++ b/contrib/jenkins/jobs/os-board-vm-matrix/config.xml @@ -0,0 +1,132 @@ + + + + Shim of a job to build all production VM variants. This is independent from os-board-vm because I'm still figuring this stuff out + + 30 + 50 + -1 + -1 + + false + + + + + MANIFEST_URL + + https://github.com/coreos/manifest-builds.git + + + MANIFEST_REF + + refs/tags/ + + + MANIFEST_NAME + + release.xml + + + COREOS_OFFICIAL + + + + 0 + 1 + + + + + + + false + false + + + + true + false + false + false + + true + + + FORMAT + + qemu + qemu_uefi + ami + ami_vmdk + pxe + iso + openstack + rackspace + rackspace_onmetal + rackspace_vhd + vagrant + vagrant_vmware_fusion + virtualbox + vmware + vmware_ova + vmware_insecure + xen + gce + brightbox + cloudstack + cloudstack_vhd + digitalocean + exoscale + azure + hyperv + niftycloud + cloudsigma + packet + interoute + + + + GROUP + + developer + alpha + beta + stable + + + + BOARD + + amd64-usr + arm64-usr + + + + (BOARD == "amd64-usr" || ( BOARD == "arm64-usr" && (FORMAT == "qemu_uefi" || FORMAT == "pxe") )) && ((COREOS_OFFICIAL == "0" && GROUP == "developer") || (COREOS_OFFICIAL == "1" && GROUP != "developer")) + + + + + + + + FORMAT=${FORMAT} +GROUP=${GROUP} +BOARD=${BOARD} + + + os-board-vm + ALWAYS + false + false + + + + + + + + false + + diff --git a/contrib/jenkins/jobs/os-board-vm/config.xml b/contrib/jenkins/jobs/os-board-vm/config.xml new file mode 100644 index 0000000000..2ccc1d5e7f --- /dev/null +++ b/contrib/jenkins/jobs/os-board-vm/config.xml @@ -0,0 +1,209 @@ + + + + + + 30 + 50 + -1 + -1 + + false + + + + + BOARD + Target board to build + + + amd64-usr + arm64-usr + + + + + GROUP + Update group the image should use + + + developer + alpha + beta + stable + + + + + FORMAT + VM platform or image type to generate. + + + qemu + qemu_uefi + ami + ami_vmdk + pxe + iso + openstack + qemu_uefi + rackspace + rackspace_onmetal + rackspace_vhd + vagrant + vagrant_vmware_fusion + virtualbox + vmware + vmware_ova + vmware_insecure + xen + gce + brightbox + cloudstack + cloudstack_vhd + digitalocean + exoscale + azure + hyperv + niftycloud + cloudsigma + packet + interoute + + + + + MANIFEST_URL + + https://github.com/coreos/manifest-builds.git + + + MANIFEST_REF + + refs/tags/ + + + MANIFEST_NAME + + release.xml + + + COREOS_OFFICIAL + + + + 0 + 1 + + + + + + + false + false + + + + + coreos + false + false + false + false + + true + + + os-mantle + + + + + false + + + #!/bin/bash -ex + +# build may not be started without a ref value +[[ -n "${MANIFEST_REF#refs/tags/}" ]] + +# check that the matrix didn't go bananas +if [[ "${COREOS_OFFICIAL}" -eq 1 ]]; then + [[ "${GROUP}" != developer ]] +else + [[ "${GROUP}" == developer ]] +fi + +script() { + local script="/mnt/host/source/src/scripts/${1}"; shift + ./bin/cork enter --experimental -- "${script}" "$@" +} + +enter() { + ./bin/cork enter --experimental -- "$@" +} + +./bin/cork update --create --downgrade-replace --verify --verbose \ + --manifest-url "${MANIFEST_URL}" \ + --manifest-branch "${MANIFEST_REF}" \ + --manifest-name "${MANIFEST_NAME}" + +source .repo/manifests/version.txt +export COREOS_BUILD_ID + +# Set up GPG for signing images +export GNUPGHOME="${PWD}/.gnupg" +rm -rf "${GNUPGHOME}" +trap "rm -rf '${GNUPGHOME}'" EXIT +mkdir --mode=0700 "${GNUPGHOME}" +gpg --import "${GPG_SECRET_KEY_FILE}" + +if [[ "${GROUP}" == developer ]]; then + root="gs://builds.developer.core-os.net" + dlroot="" +else + root="gs://builds.release.core-os.net/${GROUP}" + dlroot="--download_root https://${GROUP}.release.core-os.net" +fi + +mkdir -p tmp +./bin/cork download-image --root="${root}/boards/${BOARD}/${COREOS_VERSION}" \ + --json-key="${GOOGLE_APPLICATION_CREDENTIALS}" \ + --cache-dir=./tmp \ + --platform=qemu +enter lbunzip2 -k -f /mnt/host/source/tmp/coreos_production_image.bin.bz2 + +script image_to_vm.sh --board=${BOARD} \ + --format=${FORMAT} \ + --prod_image \ + --getbinpkg \ + --getbinpkgver=${COREOS_VERSION} \ + --from=/mnt/host/source/tmp/ \ + --to=/mnt/host/source/tmp/ \ + --sign=buildbot@coreos.com \ + --sign_digests=buildbot@coreos.com \ + --upload_root="${root}" \ + --upload ${dlroot} + + + + + chroot/build/${BOARD}/var/lib/portage/pkgs/*/*.tbz2,chroot/var/lib/portage/pkgs/*/*.tbz2,tmp/* + + + + + + + 9de82aed-ae50-4233-989f-d454b0449b96 + GOOGLE_APPLICATION_CREDENTIALS + + + 12e071a3-694f-40e8-8595-489236d943cc + GPG_SECRET_KEY_FILE + + + + + diff --git a/contrib/jenkins/jobs/os-kola-qemu/config.xml b/contrib/jenkins/jobs/os-kola-qemu/config.xml new file mode 100644 index 0000000000..8044d7b35e --- /dev/null +++ b/contrib/jenkins/jobs/os-kola-qemu/config.xml @@ -0,0 +1,152 @@ + + + + + + 30 + 50 + -1 + -1 + + false + + + os-kola-qemu/ + + + + + BOARD + Target board to build + + + amd64-usr + arm64-usr + + + + + MANIFEST_URL + + https://github.com/coreos/manifest-builds.git + + + MANIFEST_REF + + refs/tags/ + + + MANIFEST_NAME + + release.xml + + + COREOS_OFFICIAL + + + + 0 + 1 + + + + + + + false + false + + + + coreos && kvm + false + false + false + false + + true + + + os-mantle + + + + + false + + + #!/bin/bash -ex + +# build may not be started without a ref value +[[ -n "${MANIFEST_REF#refs/tags/}" ]] + +enter() { + ./bin/cork enter --experimental -- "$@" +} + +script() { + local script="/mnt/host/source/src/scripts/${1}"; shift + enter "${script}" "$@" +} + +./bin/cork update --create --downgrade-replace --verify --verbose \ + --manifest-url "${MANIFEST_URL}" \ + --manifest-branch "${MANIFEST_REF}" \ + --manifest-name "${MANIFEST_NAME}" +source .repo/manifests/version.txt + +if [[ "${COREOS_OFFICIAL}" -eq 1 ]]; then + root="gs://builds.release.core-os.net/stable" +else + root="gs://builds.developer.core-os.net" +fi + +mkdir -p tmp +./bin/cork download-image --root="${root}/boards/${BOARD}/${COREOS_VERSION}" \ + --json-key="${GOOGLE_APPLICATION_CREDENTIALS}" \ + --cache-dir=./tmp \ + --platform=qemu +enter lbunzip2 -k -f /mnt/host/source/tmp/coreos_production_image.bin.bz2 + +if [[ "${BOARD}" == arm64* ]]; then + script setup_board --board=${BOARD} \ + --getbinpkgver="${COREOS_VERSION}" \ + --regen_configs_only + enter emerge-arm64-usr --nodeps -qugKN sys-firmware/edk2-armvirt +fi + +enter sudo kola run --board="${BOARD}" \ + --parallel=2 \ + --qemu-image="/mnt/host/source/tmp/coreos_production_image.bin" \ + --tapfile="/mnt/host/source/tmp/${JOB_NAME##*/}.tap" + + + + + tmp/*.tap + true + true + true + true + false + false + true + true + true + true + + + tmp/*,chroot/var/lib/portage/pkgs/*/*.tbz2 + + + + + + + 9de82aed-ae50-4233-989f-d454b0449b96 + GOOGLE_APPLICATION_CREDENTIALS + + + + + diff --git a/contrib/jenkins/jobs/os-manifest/config.xml b/contrib/jenkins/jobs/os-manifest/config.xml new file mode 100644 index 0000000000..31bf819014 --- /dev/null +++ b/contrib/jenkins/jobs/os-manifest/config.xml @@ -0,0 +1,214 @@ + + + + + + 30 + 50 + -1 + -1 + + false + + + https://github.com/coreos/manifest/ + + + + + MANIFEST_REF + Branch or tag to build + master + + + LOCAL_MANIFEST + Amend the checked in manifest + +https://wiki.cyanogenmod.org/w/Doc:_Using_manifests#The_local_manifest + + + + + + + os-* + + + + false + false + + + + + 2 + + + origin + https://github.com/coreos/manifest.git + + + + + ${MANIFEST_REF} + + + false + + https://github.com/coreos/manifest + + + + + manifest + + + + + coreos + false + false + false + false + + false + + + os-mantle + + + + + true + UseGlobalSetting + true + + false + + + #!/bin/bash -ex + +COREOS_OFFICIAL=0 + +finish() { + local tag="$1" + git -C "${WORKSPACE}/manifest" push \ + "ssh://git@github.com/coreos/manifest-builds.git" \ + "refs/tags/${tag}:refs/tags/${tag}" + tee "${WORKSPACE}/manifest.properties" <<EOF +MANIFEST_URL = https://github.com/coreos/manifest-builds.git +MANIFEST_REF = refs/tags/${tag} +MANIFEST_NAME = release.xml +COREOS_OFFICIAL = ${COREOS_OFFICIAL:-0} +EOF +} + +# Branches are of the form remote-name/branch-name. Tags are just tag-name. +# If we have a release tag use it, for branches we need to make a tag. +if [[ "${GIT_BRANCH}" != */* ]]; then + COREOS_OFFICIAL=1 + finish "${GIT_BRANCH}" + exit +fi + +MANIFEST_BRANCH="${GIT_BRANCH##*/}" +MANIFEST_NAME="${MANIFEST_BRANCH}.xml" +[[ -f "manifest/${MANIFEST_NAME}" ]] + +source manifest/version.txt +export COREOS_BUILD_ID="${MANIFEST_BRANCH}-${BUILD_NUMBER}" + +# hack to get repo to set things up using the manifest repo we already have +# (amazing that it tolerates this considering it usually is so intolerant) +mkdir -p .repo +ln -sfT ../manifest .repo/manifests +ln -sfT ../manifest/.git .repo/manifests.git + +# Cleanup/setup local manifests +rm -rf .repo/local_manifests +if [[ -n "${LOCAL_MANIFEST}" ]]; then + mkdir -p .repo/local_manifests + cat >.repo/local_manifests/local.xml <<<"${LOCAL_MANIFEST}" +fi + +./bin/cork update --create --downgrade-replace --verbose \ + --manifest-url "${GIT_URL}" \ + --manifest-branch "${GIT_COMMIT}" \ + --manifest-name "${MANIFEST_NAME}" \ + --new-version "${COREOS_VERSION}" \ + --sdk-version "${COREOS_SDK_VERSION}" + +./bin/cork enter --experimental -- sh -c \ + "pwd; repo manifest -r > '/mnt/host/source/manifest/${COREOS_BUILD_ID}.xml'" + +cd manifest +git add "${COREOS_BUILD_ID}.xml" + +ln -sf "${COREOS_BUILD_ID}.xml" default.xml +ln -sf "${COREOS_BUILD_ID}.xml" release.xml +git add default.xml release.xml + +tee version.txt <<EOF +COREOS_VERSION=${COREOS_VERSION_ID}+${COREOS_BUILD_ID} +COREOS_VERSION_ID=${COREOS_VERSION_ID} +COREOS_BUILD_ID=${COREOS_BUILD_ID} +COREOS_SDK_VERSION=${COREOS_SDK_VERSION} +EOF +git add version.txt + +EMAIL="jenkins@jenkins.coreos.systems" +GIT_AUTHOR_NAME="CoreOS Jenkins" +GIT_COMMITTER_NAME="${GIT_AUTHOR_NAME}" +export EMAIL GIT_AUTHOR_NAME GIT_COMMITTER_NAME +git commit \ + -m "${COREOS_BUILD_ID}: add build manifest" \ + -m "Based on ${GIT_URL} branch ${MANIFEST_BRANCH}" \ + -m "${BUILD_URL}" +git tag -m "${COREOS_BUILD_ID}" "${COREOS_BUILD_ID}" HEAD + +# assert that what we just did will work, update symlink because verify doesn't have a --manifest-name option yet +cd "${WORKSPACE}" +ln -sf "manifests/${COREOS_BUILD_ID}.xml" .repo/manifest.xml +./bin/cork verify + +finish "${COREOS_BUILD_ID}" + + + + + true + + + manifest.properties + false + false + false + true + + + + + + + manifest.properties + true + false + false + + + os-toolchains,os-sdk + UNSTABLE_OR_BETTER + false + + + + + + + + 8b41f1e0-1ea9-4dc5-b64f-9b3b883b78c2 + + false + + + diff --git a/contrib/jenkins/jobs/os-mantle/config.xml b/contrib/jenkins/jobs/os-mantle/config.xml new file mode 100644 index 0000000000..5c44d8b329 --- /dev/null +++ b/contrib/jenkins/jobs/os-mantle/config.xml @@ -0,0 +1,75 @@ + + + + Build mantle binaries from master to be consumed as artifacts. + + 20 + 30 + -1 + -1 + + false + + + https://github.com/coreos/mantle/ + + + + * + + + + false + false + + + + 2 + + + origin + https://github.com/coreos/mantle/ + + + + + refs/remotes/origin/master + + + false + + + + coreos + false + false + false + false + + + H/15 * * * * + false + + + false + + + docker run --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp -e GOMAXPROCS=4 golang:1.6.2 ./build + + + + docker run --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp -e GOMAXPROCS=4 golang:1.6.2 ./test + + + + + + bin/* + false + true + true + true + + + + diff --git a/contrib/jenkins/jobs/os-sdk/config.xml b/contrib/jenkins/jobs/os-sdk/config.xml new file mode 100644 index 0000000000..451c816a1b --- /dev/null +++ b/contrib/jenkins/jobs/os-sdk/config.xml @@ -0,0 +1,134 @@ + + + + Build SDK tarball and binary packages. + + 30 + 50 + 3 + 3 + + false + + + + + USE_CACHE + Enable use of any binary packages cached locally from previous builds. + true + + + MANIFEST_URL + + https://github.com/coreos/manifest-builds.git + + + MANIFEST_REF + + refs/tags/ + + + MANIFEST_NAME + + release.xml + + + COREOS_OFFICIAL + + + + 0 + 1 + + + + + + + false + false + + + + coreos + false + false + false + false + + false + + + os-mantle + + + + + true + UseGlobalSetting + true + + false + + + #!/bin/bash -ex + +# build may not be started without a ref value +[[ -n "${MANIFEST_REF#refs/tags/}" ]] + +enter() { + ./bin/cork enter --experimental -- "$@" +} + +# hack because catalyst leaves things chowned as root +[[ -d .cache/sdks ]] && sudo chown -R $USER .cache/sdks + +./bin/cork update --create --downgrade-replace --verify --verbose \ + --manifest-url "${MANIFEST_URL}" \ + --manifest-branch "${MANIFEST_REF}" \ + --manifest-name "${MANIFEST_NAME}" + +source .repo/manifests/version.txt +export COREOS_BUILD_ID + +# Set up GPG for signing images +export GNUPGHOME="${PWD}/.gnupg" +rm -rf "${GNUPGHOME}" +trap "rm -rf '${GNUPGHOME}'" EXIT +mkdir --mode=0700 "${GNUPGHOME}" +gpg --import "${GPG_SECRET_KEY_FILE}" + +# Wipe all of catalyst or just clear out old tarballs taking up space +if [ "$USE_CACHE" = false ]; then + sudo rm -rf src/build/catalyst +fi +sudo rm -rf src/build/catalyst/builds + +S=/mnt/host/source/src/scripts +enter ${S}/update_chroot +enter sudo emerge -uv --jobs=2 catalyst +enter sudo ${S}/bootstrap_sdk \ + --sign buildbot@coreos.com --sign_digests buildbot@coreos.com \ + --upload --upload_root gs://builds.developer.core-os.net + + + + + src/build/catalyst/packages/coreos-sdk/**/*.tbz2,chroot/var/lib/portage/pkgs/*/*.tbz2 + + + + + + + 12e071a3-694f-40e8-8595-489236d943cc + GPG_SECRET_KEY_FILE + + + 9de82aed-ae50-4233-989f-d454b0449b96 + GOOGLE_APPLICATION_CREDENTIALS + + + + + diff --git a/contrib/jenkins/jobs/os-toolchains/config.xml b/contrib/jenkins/jobs/os-toolchains/config.xml new file mode 100644 index 0000000000..1f83b867db --- /dev/null +++ b/contrib/jenkins/jobs/os-toolchains/config.xml @@ -0,0 +1,162 @@ + + + + Build cross compilers for all target boards. + + 30 + 50 + 3 + 3 + + false + + + + + USE_CACHE + Enable use of any binary packages cached locally from previous builds. + true + + + MANIFEST_URL + + https://github.com/coreos/manifest-builds.git + + + MANIFEST_REF + + refs/tags/ + + + MANIFEST_NAME + + release.xml + + + COREOS_OFFICIAL + + + + 0 + 1 + + + + + + + false + false + + + + coreos + false + false + false + false + + false + + + os-mantle + + + + + true + UseGlobalSetting + true + + false + + + #!/bin/bash -ex + +# build may not be started without a ref value +[[ -n "${MANIFEST_REF#refs/tags/}" ]] + +enter() { + ./bin/cork enter --experimental -- "$@" +} + +# hack because catalyst leaves things chowned as root +[[ -d .cache/sdks ]] && sudo chown -R $USER .cache/sdks + +./bin/cork update --create --downgrade-replace --verify --verbose \ + --manifest-url "${MANIFEST_URL}" \ + --manifest-branch "${MANIFEST_REF}" \ + --manifest-name "${MANIFEST_NAME}" + +source .repo/manifests/version.txt +export COREOS_BUILD_ID + +# Set up GPG for signing images +export GNUPGHOME="${PWD}/.gnupg" +rm -rf "${GNUPGHOME}" +trap "rm -rf '${GNUPGHOME}'" EXIT +mkdir --mode=0700 "${GNUPGHOME}" +gpg --import "${GPG_SECRET_KEY_FILE}" + +# Wipe all of catalyst or just clear out old tarballs taking up space +if [ "$USE_CACHE" = false ]; then + sudo rm -rf src/build/catalyst +fi +sudo rm -rf src/build/catalyst/builds + +S=/mnt/host/source/src/scripts +enter sudo emerge -uv --jobs=2 catalyst +enter sudo ${S}/build_toolchains \ + --sign buildbot@coreos.com --sign_digests buildbot@coreos.com \ + --upload --upload_root gs://builds.developer.core-os.net + + + + + false + + + src/build/catalyst/packages/coreos-toolchains/**/*.tbz2,chroot/var/lib/portage/pkgs/*/*.tbz2 + + + + + + + + BOARD=amd64-usr + + + os-board-packages + UNSTABLE_OR_BETTER + false + + + + + + BOARD=arm64-usr + + + os-board-packages + UNSTABLE_OR_BETTER + false + + + + + + + + + 12e071a3-694f-40e8-8595-489236d943cc + GPG_SECRET_KEY_FILE + + + 9de82aed-ae50-4233-989f-d454b0449b96 + GOOGLE_APPLICATION_CREDENTIALS + + + + + diff --git a/contrib/jenkins/jobs/os-watch-coreos-overlay/config.xml b/contrib/jenkins/jobs/os-watch-coreos-overlay/config.xml new file mode 100644 index 0000000000..cf54f6773d --- /dev/null +++ b/contrib/jenkins/jobs/os-watch-coreos-overlay/config.xml @@ -0,0 +1,125 @@ + + + + + + 5 + 5 + -1 + -1 + + false + + + https://github.com/coreos/coreos-overlay/ + + + + os-* + + + + false + true + + + + + true + + + + + + 2 + + + origin + https://github.com/coreos/coreos-overlay.git + + + + + origin/master + + + origin/build-* + + + false + + https://github.com/coreos/coreos-overlay + + + + + coreos-overlay + + + + + coreos + false + false + false + false + + + H/15 * * * * + false + + + false + + + #!/bin/bash -ex + +# This all assumes the manifest branch is the same as the +# scripts branch, e.g. "master" or "build-123" and so on. +UPSTREAM_REF="refs/heads/${GIT_BRANCH#origin/}" +MANIFEST_REF="${UPSTREAM_REF}" + +cat >coreos-overlay.properties <<EOF +MANIFEST_REF = ${MANIFEST_REF} +LOCAL_MANIFEST = <?xml version="1.0" encoding="UTF-8"?>\ + <manifest>\ + <remove-project name="coreos/coreos-overlay" />\ + <project name="coreos/coreos-overlay" \ + path="src/third_party/coreos-overlay" \ + revision="${GIT_COMMIT}" \ + upstream="${UPSTREAM_REF}" />\ + </manifest> +EOF + + + + + true + + + coreos-overlay.properties + false + false + false + true + + + + + + + coreos-overlay.properties + true + false + false + + + os-manifest, + UNSTABLE_OR_BETTER + false + + + + + + diff --git a/contrib/jenkins/jobs/os-watch-portage-stable/config.xml b/contrib/jenkins/jobs/os-watch-portage-stable/config.xml new file mode 100644 index 0000000000..89245bb027 --- /dev/null +++ b/contrib/jenkins/jobs/os-watch-portage-stable/config.xml @@ -0,0 +1,125 @@ + + + + + + 5 + 5 + -1 + -1 + + false + + + https://github.com/coreos/portage-stable/ + + + + os-* + + + + false + true + + + + + true + + + + + + 2 + + + origin + https://github.com/coreos/portage-stable.git + + + + + origin/master + + + origin/build-* + + + false + + https://github.com/coreos/portage-stable + + + + + portage-stable + + + + + coreos + false + false + false + false + + + H/15 * * * * + false + + + false + + + #!/bin/bash -ex + +# This all assumes the manifest branch is the same as the +# scripts branch, e.g. "master" or "build-123" and so on. +UPSTREAM_REF="refs/heads/${GIT_BRANCH#origin/}" +MANIFEST_REF="${UPSTREAM_REF}" + +cat >portage-stable.properties <<EOF +MANIFEST_REF = ${MANIFEST_REF} +LOCAL_MANIFEST = <?xml version="1.0" encoding="UTF-8"?>\ + <manifest>\ + <remove-project name="coreos/portage-stable" />\ + <project name="coreos/portage-stable" \ + path="src/third_party/portage-stable" \ + revision="${GIT_COMMIT}" \ + upstream="${UPSTREAM_REF}" />\ + </manifest> +EOF + + + + + true + + + portage-stable.properties + false + false + false + true + + + + + + + portage-stable.properties + true + false + false + + + os-manifest, + UNSTABLE_OR_BETTER + false + + + + + + diff --git a/contrib/jenkins/jobs/os-watch-scripts/config.xml b/contrib/jenkins/jobs/os-watch-scripts/config.xml new file mode 100644 index 0000000000..53f039be3b --- /dev/null +++ b/contrib/jenkins/jobs/os-watch-scripts/config.xml @@ -0,0 +1,125 @@ + + + + + + 5 + 5 + -1 + -1 + + false + + + https://github.com/coreos/scripts/ + + + + os-* + + + + false + true + + + + + true + + + + + + 2 + + + origin + https://github.com/coreos/scripts.git + + + + + origin/master + + + origin/build-* + + + false + + https://github.com/coreos/scripts + + + + + scripts + + + + + coreos + false + false + false + false + + + H/15 * * * * + false + + + false + + + #!/bin/bash -ex + +# This all assumes the manifest branch is the same as the +# scripts branch, e.g. "master" or "build-123" and so on. +UPSTREAM_REF="refs/heads/${GIT_BRANCH#origin/}" +MANIFEST_REF="${UPSTREAM_REF}" + +cat >scripts.properties <<EOF +MANIFEST_REF = ${MANIFEST_REF} +LOCAL_MANIFEST = <?xml version="1.0" encoding="UTF-8"?>\ + <manifest>\ + <remove-project name="coreos/scripts" />\ + <project name="coreos/scripts" \ + path="src/scripts" \ + revision="${GIT_COMMIT}" \ + upstream="${UPSTREAM_REF}" />\ + </manifest> +EOF + + + + + true + + + scripts.properties + false + false + false + true + + + + + + + scripts.properties + true + false + false + + + os-manifest, + UNSTABLE_OR_BETTER + false + + + + + +