Question

Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

2.2.5. Worker Setup

2.2.5.1. Creating a worker

Typically, you will be adding a worker to an existing buildmaster, to provide additional architecture coverage. The Buildbot administrator will give you several pieces of information necessary to connect to the buildmaster. You should also be somewhat familiar with the project being tested so that you can troubleshoot build problems locally.

Buildbot exists to make sure that the project’s stated how to build it process actually works. To this end, the worker should run in an environment just like that of your regular developers. Typically the project’s build process is documented somewhere (README, INSTALL, etc), in a document that should mention all library dependencies and contain a basic set of build instructions. This document will be useful as you configure the host and account in which worker runs.

Here’s a good checklist for setting up a worker:

1. Set up the account

It is recommended (although not mandatory) to set up a separate user account for the worker. This account is frequently named buildbot or worker. This serves to isolate your personal working environment from that of the worker’s, and helps to minimize the security threat posed by letting possibly-unknown contributors run arbitrary code on your system. The account should have a minimum of fancy init scripts.

2. Install the Buildbot code

Follow the instructions given earlier (Installing the code). If you use a separate worker account, and you didn’t install the Buildbot code to a shared location, then you will need to install it with --home=~ for each account that needs it.

3. Set up the host

Make sure the host can actually reach the buildmaster. Usually the buildmaster is running a status webserver on the same machine, so simply point your web browser at it and see if you can get there. Install whatever additional packages or libraries the project’s INSTALL document advises. (or not: if your worker is supposed to make sure that building without optional libraries still works, then don’t install those libraries.)

Again, these libraries don’t necessarily have to be installed to a site-wide shared location, but they must be available to your build process. Accomplishing this is usually very specific to the build process, so installing them to /usr or /usr/local is usually the best approach.

4. Test the build process

Follow the instructions in the INSTALL document, in the worker’s account. Perform a full CVS (or whatever) checkout, configure, make, run tests, etc. Confirm that the build works without manual fussing. If it doesn’t work when you do it manually, it will be unlikely to work when Buildbot attempts to do it in an automated fashion.

5. Choose a base directory

This should be somewhere in the worker’s account, typically named after the project which is being tested. The worker will not touch any file outside of this directory. Something like ~/Buildbot or ~/Workers/fooproject is appropriate.

6. Get the buildmaster host/port, workername, and password

When the Buildbot admin configures the buildmaster to accept and use your worker, they will provide you with the following pieces of information:

• your worker’s name

• the password assigned to your worker

• the hostname and port number of the buildmaster

7. Create the worker

Now run the ‘worker’ command as follows:

buildbot-worker create-worker BASEDIR MASTERHOST:PORT WORKERNAME PASSWORD

This will create the base directory and a collection of files inside, including the buildbot.tac file that contains all the information you passed to the buildbot-worker command.

8. Fill in the hostinfo files

When it first connects, the worker will send a few files up to the buildmaster which describe the host that it is running on. These files are presented on the web status display so that developers have more information to reproduce any test failures that are witnessed by the Buildbot. There are sample files in the info subdirectory of the Buildbot’s base directory. You should edit these to correctly describe you and your host.

BASEDIR/info/admin should contain your name and email address. This is the worker admin address, and will be visible from the build status page (so you may wish to munge it a bit if address-harvesting spambots are a concern).

BASEDIR/info/host should be filled with a brief description of the host: OS, version, memory size, CPU speed, versions of relevant libraries installed, and finally the version of the Buildbot code which is running the worker.

The optional BASEDIR/info/access_uri can specify a URI which will connect a user to the machine. Many systems accept ssh://hostname URIs for this purpose.

If you run many workers, you may want to create a single ~worker/info file and share it among all the workers with symlinks.

Worker Options

There are a handful of options you might want to use when creating the worker with the buildbot-worker create-worker <options> DIR <params> command. You can type buildbot-worker create-worker --help for a summary. To use these, just include them on the buildbot-worker create-worker command line, like this

buildbot-worker create-worker --umask=0o22 ~/worker buildmaster.example.org:42012 \
    {myworkername} {mypasswd}

--protocol

unicode_encoding

This represents the encoding that Buildbot should use when converting unicode commandline arguments into byte strings in order to pass to the operating system when spawning new processes.

The default value is what Python’s sys.getfilesystemencoding returns, which on Windows is ‘mbcs’, on Mac OSX is ‘utf-8’, and on Unix depends on your locale settings.

If you need a different encoding, this can be changed in your worker’s buildbot.tac file by adding a unicode_encoding argument to the Worker constructor.

s = Worker(buildmaster_host, port, workername, passwd, basedir,
           keepalive, usepty, umask=umask, maxdelay=maxdelay,
           unicode_encoding='utf-8', allow_shutdown='signal')

Worker TLS Configuration

tls

See --useTls option above as an alternative to setting the conneciton_string manually.

connection_string

For TLS connections to the master, the connection_string-argument must be passed to the worker constructor. buildmaster_host and port must then be None.

connection_string will be used to create a client endpoint with clientFromString. An example of connection_string is "TLS:buildbot-master.com:9989".

See more about how to formulate the connection string in ConnectionStrings.

Example TLS connection string:

s = Worker(None, None, workername, passwd, basedir, keepalive,
           connection_string='TLS:buildbot-master.com:9989')

Make sure the worker trusts the certificate of the master. If you have a non-authoritative certificate (CA is self-signed), the trustRoots parameter can be used.

s = Worker(None, None, workername, passwd, basedir, keepalive,
           connection_string=
           'TLS:buildbot-master.com:9989:trustRoots=/dir-with-ca-certs')

It must point to a directory with PEM-encoded certificates. For example:

$ cat /dir-with-ca-certs/ca.pem
-----BEGIN CERTIFICATE-----
MIIE9DCCA9ygAwIBAgIJALEqLrC/m1w3MA0GCSqGSIb3DQEBCwUAMIGsMQswCQYD
VQQGEwJaWjELMAkGA1UECBMCUUExEDAOBgNVBAcTB05vd2hlcmUxETAPBgNVBAoT
CEJ1aWxkYm90MRkwFwYDVQQLExBEZXZlbG9wbWVudCBUZWFtMRQwEgYDVQQDEwtC
dWlsZGJvdCBDQTEQMA4GA1UEKRMHRWFzeVJTQTEoMCYGCSqGSIb3DQEJARYZYnVp
bGRib3RAaW50ZWdyYXRpb24udGVzdDAeFw0xNjA5MDIxMjA5NTJaFw0yNjA4MzEx
MjA5NTJaMIGsMQswCQYDVQQGEwJaWjELMAkGA1UECBMCUUExEDAOBgNVBAcTB05v
d2hlcmUxETAPBgNVBAoTCEJ1aWxkYm90MRkwFwYDVQQLExBEZXZlbG9wbWVudCBU
ZWFtMRQwEgYDVQQDEwtCdWlsZGJvdCBDQTEQMA4GA1UEKRMHRWFzeVJTQTEoMCYG
CSqGSIb3DQEJARYZYnVpbGRib3RAaW50ZWdyYXRpb24udGVzdDCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBALJZcC9j4XYBi1fYT/fibY2FRWn6Qh74b1Pg
I7iIde6Sf3DPdh/ogYvZAT+cIlkZdo4v326d0EkuYKcywDvho8UeET6sIYhuHPDW
lRl1Ret6ylxpbEfxFNvMoEGNhYAP0C6QS2eWEP9LkV2lCuMQtWWzdedjk+efqBjR
Gozaim0lr/5lx7bnVx0oRLAgbI5/9Ukbopansfr+Cp9CpFpbNPGZSmELzC3FPKXK
5tycj8WEqlywlha2/VRnCZfYefB3aAuQqQilLh+QHyhn6hzc26+n5B0l8QvrMkOX
atKdznMLzJWGxS7UwmDKcsolcMAW+82BZ8nUCBPF3U5PkTLO540CAwEAAaOCARUw
ggERMB0GA1UdDgQWBBT7A/I+MZ1sFFJ9jikYkn51Q3wJ+TCB4QYDVR0jBIHZMIHW
gBT7A/I+MZ1sFFJ9jikYkn51Q3wJ+aGBsqSBrzCBrDELMAkGA1UEBhMCWloxCzAJ
BgNVBAgTAlFBMRAwDgYDVQQHEwdOb3doZXJlMREwDwYDVQQKEwhCdWlsZGJvdDEZ
MBcGA1UECxMQRGV2ZWxvcG1lbnQgVGVhbTEUMBIGA1UEAxMLQnVpbGRib3QgQ0Ex
EDAOBgNVBCkTB0Vhc3lSU0ExKDAmBgkqhkiG9w0BCQEWGWJ1aWxkYm90QGludGVn
cmF0aW9uLnRlc3SCCQCxKi6wv5tcNzAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEB
CwUAA4IBAQCJGJVMAmwZRK/mRqm9E0e3s4YGmYT2jwX5IX17XljEy+1cS4huuZW2
33CFpslkT1MN/r8IIZWilxT/lTujHyt4eERGjE1oRVKU8rlTH8WUjFzPIVu7nkte
09abqynAoec8aQukg79NRCY1l/E2/WzfnUt3yTgKPfZmzoiN0K+hH4gVlWtrizPA
LaGwoslYYTA6jHNEeMm8OQLNf17OTmAa7EpeIgVpLRCieI9S3JIG4WYU8fVkeuiU
cB439SdixU4cecVjNfFDpq6JM8N6+DQoYOSNRt9Dy0ioGyx5D4lWoIQ+BmXQENal
gw+XLyejeNTNgLOxf9pbNYMJqxhkTkoE
-----END CERTIFICATE-----

Using TCP in connection_string is the equivalent to using the buildmaster_host and port arguments.

s = Worker(None, None, workername, passwd, basedir, keepalive
           connection_string='TCP:buildbot-master.com:9989')

is equivalent to

s = Worker('buildbot-master.com', 9989, workername, passwd, basedir,
           keepalive)