Buildbot Configuration Tools

These scripts make it relatively simple affair to configure a buildbot controller and a pool of workers that are sandbox-aware. While recommended, the use of sandboxes is not required, but the findFile command from that tool suite needs to be installed.

makeBuildbotManager
Creates the configuration of a buildbot cluster controller on the local host. Population of the final master.cfg file can be greatly aided by using makeBuildbotConfig.
makeBuildbotWorker
Creates the configuration of one or more buildbot workers on the local host.
makeBuildbotConfig
Reads a description of various workers, build steps and dependency ordering and adds appropriate declarations to a master.cfg prototype. This utility also generates a buildbotDependencies.dot file that can be used to visualize the resulting tree of dependent schedulers.
startBuildbotManager
Starts a previously configured buildbot controller associated with the local host.
startBuildbotWorkers
Starts previously configured buildbot workers associated with the local host.
stopBuildbotManager
Stops the buildbot controller on the local host that was previously started by startBuildbotManager.
stopBuildbotWorkers
Stops buildbot workers on the local host that were previously started by startBuildbotWorkers.
buildbot-manager.service
A systemd service definition that uses startBuildbotManager and stopBuildbotManager.
buildbot-workers.service
A systemd service definition that uses startBuildbotWorkers and stopBuildbotWorkers.
emitSSHpassword
Returns an SSH password when requested by ssh-agent.

Creating the Buildbot Manager

One host will need to act as the manager of the pool of buildbot workers. If desired, the chosen host can also perform builds or it can simply serve as a dedicated controller. Workers are prepared using the makeBuildbotWorker command, which is discussed later.

A directory tree must be chosen to hold the configuration files for the manager. One suggested possibility is to create a buildbot user and place the files under its home directory.

Running makeBuildbotManager on the selected host will prepare the needed configuration files.

usage: makeBuildbotManager [--under dir]
  --under defaults to .
NOTE: makeBuildbotConfig can be used to prepare the master.cfg

By default, the configuration files are dropped under the current working directory, but the root of the tree can be specified using the ‑‑under option. This will result the creation of a directory named {hostname}‑manager, which in turn has a manager subdirectory. Note that the ‑manager suffix on the directory name permits configuration files for local workers to be dropped under the same root directory.

The master.cfg in the {hostname}‑manager/manager directory needs to be prepared before the buildbot manager daemon can be started. Some discussion is found in the buildbot tutorials and it is perfectly reasonable to follow the instructions found there. It can, however, be a somewhat overwhelming process to create a complex setup, so the makeBuildbotConfig command can be used to populate a template file with the various workers, factories, builders, etc.

makeBuildbotConfig

The makeBuildbotConfig utility accepts a trivial description of workers, build order and steps and generates a buildbot master.cfg file. It is sandbox-aware, but it can still be used in environments that do not make use of the sandbox facilities.

usage: makeBuildbotConfig [--proto Cfg.proto] {[--dependencies] dependencyFile}
Writes annotated master.cfg to standard output.
Generates "./buildbotDependencies.dot" file.
The dependencies file consists of WORKER, BUILD-ORDER and BUILD-STEP lines.
    format: WORKER: os-arch
    format: BUILD-ORDER: project:buildType:worker dependencies
    format: BUILD-STEP: forBuilderName sandboxName [command [arg [...]]]
      with forBuilderName formatted as project:buildType:workers

Prepare Prototype master.cfg

The master.cfg prototype file is a modified master.cfg.sample that has been populated with site-specific data and three specific changes. The site-specific data includes trivial entries such as title, titleURL and buildbotURL as well as the complex entries needed for change_source.

The specific changes made to the prototype are:

  1. Remove the workers definitions and replace with ##OUTPUT‑WORKERS.
  2. Remove the schedulers definitions and replace with ##OUTPUT‑SCHEDULERS.
  3. Remove the construction of a factory object and the builders definitions. Replace these with ##OUTPUT‑FACTORIES and ##OUTPUT‑SCHEDULERS, respectively.

This will yield content that appears similar to:

####### WORKERS

# The 'workers' list defines the set of recognized workers. Each element is
# a Worker object, specifying a unique worker name and password.  The same
# worker name and password must be configured on the worker.
##OUTPUT-WORKERS

...

####### SCHEDULERS

# Configure the Schedulers, which decide how to react to incoming changes.
##OUTPUT-SCHEDULERS


####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which workers can execute them.  Note that any particular build will
# only take place on one worker.
##OUTPUT-FACTORIES

##OUTPUT-BUILDERS

Dependency List

Along with the master.cfg prototype, the makeBuildbotConfig also consumes a file describing the available workers, build operations and any dependent ordering. Four distinct types of lines are supported:

A sample dependency file appears below:

# These describe available workers
# format: WORKER: name [actualOSandCPU]
WORKER: fedora-x86_64
WORKER: fedora2-x86_64 Linux.x86_64
WORKER: fedora-aarch64
WORKER: mingw64
WORKER: el8-x86_64
WORKER: darwin-x86_64
WORKER: openbsd-x86_64
WORKER: ubuntu-s390x
#WORKER: armv7l

# These describe the dependency order of various builds and
# generate schedulers
# format: BUILD-ORDER: project:buildType:workers dependsOn
BUILD-ORDER: sandboxTools:update:fedora-x86_64
BUILD-ORDER: sandboxTools:release:fedora-x86_64 sandboxTools:update:fedora-x86_64
BUILD-ORDER: vista:update:fedora-x86_64
BUILD-ORDER: vista:release:fedora-x86_64 vista:update:fedora-x86_64
BUILD-ORDER: vista:documentation:fedora-x86_64 vista:release:fedora-x86_64
BUILD-ORDER: vista:release:mingw64 vista:documentation:fedora-x86_64
BUILD-ORDER: vista:release:el8-x86_64 vista:documentation:fedora-x86_64
BUILD-ORDER: vista:release:darwin-x86_64 vista:documentation:fedora-x86_64
BUILD-ORDER: vista:release:fedora-aarch64 vista:documentation:fedora-x86_64
BUILD-ORDER: vista:release:openbsd-x86_64 vista:documentation:fedora-x86_64
BUILD-ORDER: vista:release:ubuntu-s390x vista:release:mingw64
BUILD-ORDER: vista:package:fedora-x86_64 vista:documentation:fedora-x86_64
BUILD-ORDER: vista:package:* vista:release:=


# These generate factories
# Default command is sbmake buildType
# format: BUILD-STEP: forBuilderName sandboxName [command [arg [...]]]
# with forBuilderName formatted as project:buildType:workers
BUILD-STEP: vista:update:fedora-x86_64 bt:current sbmake update
BUILD-STEP: vista:documentation:fedora-x86_64 vista_arm
BUILD-STEP: vista:release:* vista_arm sbmake default
BUILD-STEP: vista:package:* vista_arm

BUILD-STEP: sandboxTools:update:fedora-x86_64 bt:sbtools sbmake update
BUILD-STEP: sandboxTools:release:fedora-x86_64 sandboxTools sbmake default
BUILD-STEP: sandboxTools:package fedora-x86_64 sandboxTools sbmake tar

WORKER Definition

The WORKER: statement takes one mandatory argument that specifies the name of the worker, and an optional argument that explicitly specifies the default architecture associated with the worker.

Several names are already recognized by makeBuildbotConfig.awk script and local additions can be made by adding an appropriate row to the defaultArchitecture array. That said, explicitly specifying the architecture eliminates any future requirement to reapply local modifications when an updated version of makeBuildbotConfig.awk is released.

There is no strict constraint imposed on the name for a worker, but the default convention used by makeBuildBotWorker is to construct a name that identifies the operating system and target CPU architecture. This will fail to generate unique names if more than one worker is available for a particular operating system/CPU architecture pair.

BUILD-ORDER Definition

The BUILD‑ORDER: statement describes the dependency of a build step. The first argument is the name of a build step. If only one argument is provided, the named build step has no scheduling dependencies. In contrast, the presence of an optional second argument indicates the build step upon which the step named by the first argument is dependent.

Build steps are named using the convention of {projectName}:{buildType}:{worker}. The {projectName} is intended to correspond with the project name assigned to a change_source, such as an SVNPoller or GitPoller object. The {projectName} is used to select the appropriate ChangeFilter when creating a SingleBranchScheduler for the build steps with no dependencies. Build steps with dependencies are free to use an arbitrary name for the {project} portion of the build step name; however, remaining consistent with the dependent project allows wildcarding to be used in some rules.

The {worker} component for the first argument is usually a single worker, but it can also be a comma-separated list of workers. If the optional second argument is provided to specify a dependency, it may also be specified as the wildcard "*". Under those conditions, it will automatically generate names corresponding to workers for which no prior BUILD‑ORDER: definition has been made and matches the corresponding worker in the dependency.

The {worker} component of the second argument may be specified as "=", which will cause the corresponding worker name to be substituted from the first argument. Given the following fragment:

BUILD-ORDER: proj1:update:osc-arch3
BUILD-ORDER: proj1:build:osc-arch3 proj1:update:osc-arch3
BUILD-ORDER: proj1:build:osb-arch2,osa-arch1 proj1:build:osc-arch3
BUILD-ORDER: proj1:package:* proj1:build:=
BUILD-ORDER: proj1:test:osb-arch2,osa-arch1 proj1:package:=
BUILD-ORDER: proj1:test:* proj1:build:=

The third line above generates 2 dependencies: proj1:build:osb‑arch2 and project1:build:osa‑arch1 both depend upon proj1:build:osc‑arch3. In contrast, the fourth line expands to 3 dependencies: proj1:package:osa‑arch1 depends on proj1:build:osa‑arch1, proj1:package:osb‑arch2 depends on proj1:build:osb‑arch2 and proj1:package:osc‑arch3 depends on proj1:build:osc‑arch3.

While the fifth line also uses the "=" expansion, only two dependencies are generated because the list of workers is explicit: proj1:test:osa‑arch1 depends on proj1:package:osa‑arch1 and proj1:test:osb‑arch2 depends on proj1:package:osb‑arch2.

Despite the use of the "*" wildcard, the sixth line only generates a single dependency: proj1:test:osc‑arch3 depends on proj1:build:osc‑arch3. This is because entries were already defined for proj1:test:osa‑arch1 and proj1:test:osb‑arch2 by the fifth line; otherwise the expected three dependencies would have been generated.

BUILD-STEP Definition

The BUILD‑STEP: statement describes the operation to perform for a build-step. By default, these are sandbox-aware. The first argument is the name of the build-step. It would correspond to something defined by a BUILD‑ORDER: statement. Similar to the BUILD‑ORDER: statement, the worker component of the step name can be a comma-separated list or the "*" wildcard.

The second argument is the name of the associated sandbox or backing tree. Backing trees can be addresed by prefixing the backing tree name with "bt:". The special name "NOT_SB" is recognized as indicating that the command should not be executed within the context of a sandbox.

Any remaining arguments are treated as the command to invoke and its corresponding arguments. If a command is not specified as the third argument, it defaults to sbmake. If no arguments are specified for the command, one is generated from the build type component of the step name. Thus:

BUILD-STEP: vista:update:fedora-x86_64 bt:current

generates a factory to invoke sbmake update on the current backing tree. The corresponding code fragment is illustrated below:

vista_update_factory = util.BuildFactory()
vista_update_factory.addStep(steps.ShellCommand(command=[
        "workon_bt", "--bt", "current",
        "--ta", util.Interpolate("%(prop:BB_ARCH)s+release"),
        "-c", "sbmake", "update"]))

Dependency Tree Graph

The makeBuildbotConfig utility generates a buildbotDependencies.dot file as a side-effect. This is intended to help visualize the resulting dependency tree of build steps. The dot command can be used to generate a variety of output formats, but a simple SVG file might be the most common choice:

dot -Tsvg buildbotDependencies.dot > schedulerTree.svg

Creating Buildbot Workers

The makeBuildbotWorker command can be used to create worker processes on build machines. Similar to makeBuildbotManager, it can be helpful to create a dedicated user, such as buildbot, and create the configuration files under that home directory. The naming conventions are intentionally designed to support using a common directory shared across multiple hosts via a networked filesystems, so diskless compute nodes are viable candidates for build machines.

usage: makeBuildbotWorker [--under dir] [--manager host] [--password pw] [--user userLogin] [--fullname emailFromName] [--email emailAddress] workerName [...]
  --under defaults to .
  --manager defaults to ...
  --password defaults to bb-passwd
  --user defaults to ...
Default values for userLogin, emailFromName and emailAddress  will be automatically determined if not specified.
NOTE:  normal usage is to simply pass "default" as the workerName
  and an appropriate name will be generated.
Supported environment variables:
  BUILDBOT_MANAGER - hostname of buildbot manager
  BUILDBOT_DEFAULT_PW - default password for buildbot workers
    

Typical usage is to just invoke makeBuildbotWorker with default as the worker name, but one can specify a name of their own construction. If "default" is specified, the worker name is generated from the operating system type and the CPU architecture. If the name provided ends in a hyphen, it will be used almost as is: worker is appended to the provided name to construct the full name of the worker. If a hyphen does not appear anywhere in the provided name, then {arch}‑worker is appended to the provided name, where {arch} is the local CPU architecture.

Starting and Stopping Buildbot Daemons

If invoked on the controlling node, the startBuildbotManager command will start the buildbot manager daemon. It can be shutdown by invoking stopBuildbotManager on the same node.

In a similar fashion, startBuildbotWorkers and stopBuildbotWorkers can be used to start or stop all of the workers associated with the local host.

Environment variables can be set in buildbotService.env file, which is typically placed in /etc/sysconfig. It consists of at least these variables:

BUILDBOT_USER
Specifies the login name of the user to run the buildbot-related processes.
BUILDBOT_ROOT
Specifies the root of the directory tree holding the configuration files for the local workers and/or manager.
BUILDBOT_BIN
Specifies the directory in which the start and stop scripts have been installed. This may very well be identical to BUILDBOT_ROOT.

An example buildbotService.env is below:

# variables for local Buildbot configuration
BUILDBOT_BIN=/usr/local/bin
BUILDBOT_ROOT=/storage/Buildbot
BUILDBOT_USER=buildbot

Systemd Services

Two service unit files are provided to start and stop buildbot components: buildbot-manager.service and buildbot-workers.service.

On the appropriate hosts, these are normally copied to /etc/systemd/system. They default to running their respective daemons as the user buildbot. This can be changed by either directly editing the *.service files and altering the User entry in the [Service] section or creating an appropriate local.conf file. The latter approach has the benefit that updated buildbot-manager.service and buildbot-workers.service files can be deployed without requiring modification but takes a slight amount of additional effort to setup this up for the first time.

The steps to deploy a local.conf file are straightforward. The local.conf file will be stored in a directory whose name is constructed by appending ".d" to the end of the unit file's name. For example, this would create the override to have the manager run as the user svn:

# cd /etc/systemd/system
# mkdir buildbot-manager.service.d
# cd buildbot-manager.service.d
# cat > local.conf
[Service]
User=svn
<Cntrl-D>

Once any local overrides are in place, a service can be installed by doing the appropriate systemctl enable commands and starting the services using systemctl start:

# systemctl enable buildbot-manager.service
# systemctl enable buildbot-workers.service
# systemctl start buildbot-manager
# systemctl start buildbot-workers