Bazel Remote Cache + Remote Execution in Kubernetes
![]() | ![]() | ![]() |
Bazel | REAPI | Kubernetes |
buildkube uses rules_docker and rules_k8s to build and deploy
bazel-buildfarm (java), bazel-buildbarn (golang) and/or buildgrid (python)
into an existing kubernetes cluster. These are the 3 known open-source
server-side implementations of the remote-execution-api (REAPI), plus the
closed source google Remote Build Execution
(RBE) service
(alpha).
Known clients of the REAPI include bazel
itself, recc, and possibly
pants.
WORKSPACE
file k8s_defaults
rule to point to your kubernetes$ kubectl config current-context
)$ (cd farm/ && make
install)
$ (cd farm/ && make port-forward)
$ make abseil_clone
$ make abseil
$ bazel test //...
.$ kubectl get pods
--all-namespaces
to see all.replicas
in the deploy.yaml
files and/or bazelrc
BuildFarm worker does not detect if server goes down. Must manually kubectl
delete pod --selector=k8s-app=worker
when re-installing or updating server
deployment.
When a worker registers itself with the server (operation-queue), it provides
a dict of key:value pairs that must match the action execution requirements.
In particular, the worker.config
container-image
key MUST be exactly
matching the rbe_ubuntu image tag.
/tmp/abseil-cpp/absl/utility/BUILD.bazel
1: C++
compilation of rule '//absl/utility:utility_test' failed (Exit 34). Note:
Remote connection/protocol failed with: execution failed catastrophically
. NOTE(@EdShoueten): There are three ways that can be used to alleviate this
issue:
- Spawn more workers on your cluster.
- Pass in an explicit —jobs= to the build that is the same order of magnitude
as the number of workers.- Tune this flag on the scheduler process:
https://github.com/EdSchouten/bazel-buildbarn/blob/master/cmd/bbb_scheduler/main.go#L22
main
) must match across the bazelrc
--instance_name=main
,-scheduler main|ubuntu-scheduler:8981
, and worker args bot
--remote=http://server:8980 --parent=main host-tools