Buildbot requires a connection to a database to maintain certain state information, such as tracking pending build requests. In the default configuration Buildbot uses a file-based SQLite database, stored in the state.sqlite file of the master’s base directory.
Important
SQLite3 is perfectly suitable for small setups with a few users. However, it does not scale well with large numbers of builders, workers and users. If you expect your Buildbot to grow over time, it is strongly advised to use a real database server (e.g., MySQL or Postgres).
A SQLite3 database may be migrated to a real database server using buildbot copy-db script.
See the Using A Database Server section for more details.
Override this configuration with the
For sqlite databases, since there is no host and port, relative paths are specified with sqlite:/// and absolute paths with sqlite:////. For example:
c['db_url'] = "sqlite:///state.sqlite"
SQLite requires no special configuration.
MySQL
c['db_url'] = "mysql://username:password@example.com/database_name?max_idle=300"
The max_idle argument for MySQL connections is unique to Buildbot and should be set to something less than the wait_timeout configured for your server. This controls the SQLAlchemy pool_recycle parameter, which defaults to no timeout. Setting this parameter ensures that connections are closed and re-opened after the configured amount of idle time. If you see errors such as _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away'), this means your max_idle setting is probably too high. show global variables like 'wait_timeout'; will show what the currently configured wait_timeout is on your MySQL server.
Buildbot requires use_unique=True and charset=utf8, and will add them automatically, so they do not need to be specified in db_url.
MySQL defaults to the MyISAM storage engine, but this can be overridden with the storage_engine URL argument.
Postgres
c['db_url'] = "postgresql://username:password@hostname/dbname"
PosgreSQL requires no special configuration.
Buildbot uses a message-queueing system to handle communication within the master. Messages are used to indicate events within the master, and components that are interested in those events arrange to receive them.
The message queueing implementation is configured as a dictionary in the mq option. The type key describes the type of MQ implementation to be used. Note that the implementation type cannot be changed in a reconfig.
The available implementation types are described in the following sections.
Simple
c['mq'] = {
'type' : 'simple',
'debug' : False,
}
This is the default MQ implementation. Similar to SQLite, it has no additional software dependencies, but does not support multi-master mode.
Note that this implementation also does not support message persistence across a restart of the master. For example, if a change is received, but the master shuts down before the schedulers can create build requests for it, then those schedulers will not be notified of the change when the master starts again.
The debug key, which defaults to False, can be used to enable logging of every message produced on this master.
Wamp
Note
At the moment, wamp is the only message queue implementation for multimaster. It has been privileged as this is the only message queue that has very solid support for Twisted. Other more common message queue systems like RabbitMQ (using the AMQP protocol) do not have a convincing driver for twisted, and this would require to run on threads, which will add an important performance overhead.
c['mq'] = {
'type' : 'wamp',
'router_url': 'ws://localhost:8080/ws',
'realm': 'realm1',
# valid are: none, critical, error, warn, info, debug, trace
'wamp_debug_level' : 'error'
}
This is a MQ implementation using the wamp protocol. This implementation uses Python Autobahn wamp client library, and is fully asynchronous (no use of threads). To use this implementation, you need a wamp router like Crossbar.
Please refer to Crossbar documentation for more details, but the default Crossbar setup will just work with Buildbot, provided you use the example mq configuration above, and start Crossbar with:
# of course, you should work in a virtualenv...
pip install crossbar
crossbar init
crossbar start
The implementation does not yet support wamp authentication. This MQ allows buildbot to run in multi-master mode.
Note that this implementation also does not support message persistence across a restart of the master. For example, if a change is received, but the master shuts down before the schedulers can create build requests for it, then those schedulers will not be notified of the change when the master starts again.
router_url (mandatory): points to your router websocket url.Buildbot is only supporting wamp over websocket, which is a sub-protocol of http. SSL is supported using wss:// instead of ws://.
realm (optional, defaults to buildbot): defines the wamp realm to use for your buildbot messages.
wamp_debug_level (optional, defaults to error): defines the log level of autobahn.
You must use a router with very reliable connection to the master. If for some reason, the wamp connection is lost, then the master will stop, and should be restarted via a process manager.
See Multimaster for details on the multi-master mode in Buildbot Nine.
By default, Buildbot makes coherency checks that prevent typos in your master.cfg. It makes sure schedulers are not referencing unknown builders, and enforces there is at least one builder.
In the case of an asymmetric multimaster, those coherency checks can be harmful and prevent you to implement what you want. For example, you might want to have one master dedicated to the UI, so that a big load generated by builds will not impact page load times.
To enable multi-master mode in this configuration, you will need to set the
Three basic settings describe the buildmaster in status reports:
c['title'] = "Buildbot"
c['titleURL'] = "http://buildbot.sourceforge.net/"
c['logCompressionMethod'] = 'gz'
c['logMaxSize'] = 1024*1024 # 1M
c['logMaxTailSize'] = 32768
c['logEncoding'] = 'utf-8'
The
Horizons
Previously Buildbot implemented a global configuration for horizons. Now it is implemented as a utility Builder, and shall be configured via the JanitorConfigurator.
Caches
c['caches'] = {
'Changes' : 100, # formerly c['changeCacheSize']
'Builds' : 500, # formerly c['buildCacheSize']
'chdicts' : 100,
'BuildRequests' : 10,
'SourceStamps' : 20,
'ssdicts' : 20,
'objectids' : 10,
'usdicts' : 100,
}
The
c['collapseRequests'] = True
This is a global default value for builders’
def prioritizeBuilders(buildmaster, builders):
...
c['prioritizeBuilders'] = prioritizeBuilders
By default, buildbot will attempt to start builds on builders in order, beginning with the builder with the oldest pending request. Customize this behavior with the
c['protocols'] = {"pb": {"port": 10000}}
The buildmaster will listen on a TCP port of your choosing for connections from workers. It can also use this port for connections from remote Change Sources, status clients, and debug tools. This port should be visible to the outside world, and you’ll need to tell your worker admins about your choice.
It does not matter which port you pick, as long it is externally visible; however, you should probably use something larger than 1024, since most operating systems don’t allow non-root processes to bind to low-numbered ports. If your buildmaster is behind a firewall or a NAT box of some sort, you may have to configure your firewall to permit inbound connections to this port.
c['protocols']['pb']['port'] can also be used as a connection string, as defined in the ConnectionStrings guide.
This means that you can have the buildmaster listen on a localhost-only port by doing:
c['protocols'] = {"pb": {"port": "tcp:10000:interface=127.0.0.1"}}
This might be useful if you only run workers on the same machine, and they are all configured to contact the buildmaster at localhost:10000.
connection strings can also be used to configure workers connecting over TLS. The syntax is then
c['protocols'] = {"pb": {"port":
"ssl:9989:privateKey=master.key:certKey=master.crt"}}
Please note that IPv6 addresses with : must be escaped with as well as : in paths and in paths. Read more about the connection strings format in ConnectionStrings documentation.
See also Worker TLS Configuration
The
Manhole is an interactive Python shell which allows full access to the Buildbot master instance. It is probably only useful for buildbot developers.
See documentation on Manhole implementations for available authentication and connection methods.
The manhole configuration key accepts a single instance of a Manhole class. For example:
from buildbot import manhole
c['manhole'] = manhole.PasswordManhole("tcp:1234:interface=127.0.0.1",
"admin", "passwd",
ssh_hostkey_dir="data/ssh_host_keys")
c['metrics'] = dict(log_interval=10, periodic_interval=10)
The Statistics Service (stats service for short) supports the collection of arbitrary data from within a running Buildbot instance and the export to a number of storage backends. Currently, only InfluxDB is supported as a storage backend. Also, InfluxDB (or any other storage backend) is not a mandatory dependency. Buildbot can run without it, although StatsService will be of no use in such a case. At present, StatsService can keep track of build properties, build times (start, end, duration) and arbitrary data produced inside Buildbot (more on this later).
Example usage:
captures = [stats.CaptureProperty('Builder1', 'tree-size-KiB'),
stats.CaptureBuildDuration('Builder2')]
c['services'] = []
c['services'].append(stats.StatsService(
storage_backends=[
stats.InfluxStorageService('localhost', 8086, 'root', 'root', 'test', captures)
], name="StatsService"))
The services configuration value should be initialized as a list and a StatsService instance should be appended to it as shown in the example above.
Statistics Service
- class buildbot.statistics.stats_service.StatsService
This is the main class for statistics services. It is initialized in the master configuration as shown in the example above. It takes two arguments:
storage_backendsA list of storage backends (see
- class buildbot.statistics.capture.CaptureProperty
Instance of this class declares which properties must be captured and sent to the
Advanced users can modify BuildSteps to use StatsService.yieldMetricsValue which will send arbitrary data for storage to the StatsService. It takes the following arguments:
data_nameThe name of the data being sent or storage.
post_dataA dictionary of key value pair that is sent for storage. The keys will act as columns in a database and the value is stored under that column.
buildidThe integer build id of the current build. Obtainable in all BuildSteps.
Along with using yieldMetricsValue, the user will also need to use the CaptureData capture class. As an example, we can add the following to a build step:
yieldMetricsValue('test_data_name', {'some_data': 'some_value'}, buildid)
Then, we can add in the master configuration a capture class like this:
captures = [CaptureBuildData('test_data_name', 'Builder1')]
Pass this captures list to a storage backend (as shown in the example at the top of this section) for capturing this data.
Storage Backends
Storage backends are responsible for storing any statistics data sent to them. A storage backend will generally be some sort of a database-server running on a machine. (Note: This machine may be different from the one running BuildMaster)
Currently, only InfluxDB is supported as a storage backend.
- class buildbot.statistics.storage_backends.influxdb_client.InfluxStorageService
This class is a Buildbot client to the InfluxDB storage backend. InfluxDB is a distributed, time series database that employs a key-value pair storage system.
It requires the following arguments:
urlThe URL where the service is running.
portThe port on which the service is listening.
userUsername of a InfluxDB user.
passwordPassword for user.
dbThe name of database to be used.
capturesA list of objects of
See Secret Management for details on secret concepts.
Example usage:
c['secretsProviders'] = [ .. ]
secretsProviders is a list of secrets storage. See Secret Management to configure a secret storage provider.
Since buildbot 0.9.0, buildbot has a simple feature which sends usage analysis info to buildbot.net. This is very important for buildbot developers to understand how the community is using the tools. This allows to better prioritize issues, and understand what plugins are actually being used. This will also be a tool to decide whether to keep support for very old tools. For example buildbot contains support for the venerable CVS, but we have no information whether it actually works beyond the unit tests. We rely on the community to test and report issues with the old features.
With BuildbotNetUsageData, we can know exactly what combination of plugins are working together, how much people are customizing plugins, what versions of the main dependencies people run.
We take your privacy very seriously.
BuildbotNetUsageData will never send information specific to your Code or Intellectual Property. No repository url, shell command values, host names, ip address or custom class names. If it does, then this is a bug, please report.
We still need to track unique number for installation. This is done via doing a sha1 hash of master’s hostname, installation path and fqdn. Using a secure hash means there is no way of knowing hostname, path and fqdn given the hash, but still there is a different hash for each master.
You can see exactly what is sent in the master’s twisted.log. Usage data is sent every time the master is started.
BuildbotNetUsageData can be configured with 4 values:
c['buildbotNetUsageData'] = None disables the feature
c['buildbotNetUsageData'] = 'basic' sends the basic information to buildbot including:
versions of buildbot, python and twisted
platform information (CPU, OS, distribution, python flavor (i.e CPython vs PyPy))
mq and database type (mysql or sqlite?)
www plugins usage
Plugins usages: This counts the number of time each class of buildbot is used in your configuration. This counts workers, builders, steps, schedulers, change sources. If the plugin is subclassed, then it will be prefixed with a >
example of basic report (for the metabuildbot):
{
'versions': {
'Python': '2.7.6',
'Twisted': '15.5.0',
'Buildbot': '0.9.0rc2-176-g5fa9dbf'
},
'platform': {
'machine': 'x86_64',
'python_implementation': 'CPython',
'version': '#140-Ubuntu SMP Mon Jul',
'processor':
'x86_64',
'distro:': ('Ubuntu', '14.04', 'trusty')
},
'db': 'sqlite',
'mq': 'simple',
'plugins': {
'buildbot.schedulers.forcesched.ForceScheduler': 2,
'buildbot.schedulers.triggerable.Triggerable': 1,
'buildbot.config.BuilderConfig': 4,
'buildbot.schedulers.basic.AnyBranchScheduler': 2,
'buildbot.steps.source.git.Git': 4,
'>>buildbot.steps.trigger.Trigger': 2,
'>>>buildbot.worker.base.Worker': 4,
'buildbot.reporters.irc.IRC': 1},
'www_plugins': ['buildbot_travis', 'waterfall_view']
}
c['buildbotNetUsageData'] = 'full' sends the basic information plus additional information:
{
'builders': [
['buildbot.steps.source.git.Git',
'>>>buildbot.process.buildstep.BuildStep'],
['buildbot.steps.source.git.Git',
'>>buildbot.steps.trigger.Trigger'],
['buildbot.steps.source.git.Git',
'>>>buildbot.process.buildstep.BuildStep'],
['buildbot.steps.source.git.Git',
'>>buildbot.steps.trigger.Trigger']
]
}
c['buildbotNetUsageData'] = myCustomFunction declares a callback to use to specify exactly what to send.
This custom function takes the generated data from full report in the form of a dictionary, and returns a customized report as a jsonable dictionary. You can use this to filter any information you don’t want to disclose. You can also use a custom http_proxy environment variable in order to not send any data while developing your callback.
from buildbot.plugins import util
c['user_managers'] = []
c['user_managers'].append(util.CommandlineUserManager(username="user",
passwd="userpw",
port=9990))
import re
c['validation'] = {
'branch' : re.compile(r'^[\w.+/~-]*$'),
'revision' : re.compile(r'^[ \w\.\-\/]*$'),
'property_name' : re.compile(r'^[\w\.\-\/\~:]*$'),
'property_value' : re.compile(r'^[\w\.\-\/\~:]*$'),
}
This option configures the validation applied to user inputs of various types. This validation is important since these values are often included in command-line arguments executed on workers. Allowing arbitrary input from untrusted users may raise security concerns.
The keys describe the type of input validated; the values are compiled regular expressions against which the input will be matched. The defaults for each type of input are those given in the example, above.
The
Buildbot provides two helpers for generating revision links. buildbot.revlinks.RevlinkMatcher takes a list of regular expressions and a replacement text. The regular expressions should all have the same number of capture groups. The replacement text should have sed-style references to that capture groups (i.e. ‘1’ for the first capture group), and a single ‘%s’ reference for the revision ID. The repository given is tried against each regular expression in turn. The results are then substituted into the replacement text, along with the revision ID, to obtain the revision link.
from buildbot.plugins import util
c['revlink'] = util.RevlinkMatch([r'git://notmuchmail.org/git/(.*)'],
r'http://git.notmuchmail.org/git/\1/commit/%s')
buildbot.revlinks.RevlinkMultiplexer takes a list of revision link callables, and tries each in turn, returning the first successful match.
all_repositories = {
r'https://hg/hg/mailsuite/mailclient': 'mailexe',
r'https://hg/hg/mailsuite/mapilib': 'mapilib',
r'https://hg/hg/mailsuite/imaplib': 'imaplib',
r'https://github.com/mailinc/mailsuite/mailclient': 'mailexe',
r'https://github.com/mailinc/mailsuite/mapilib': 'mapilib',
r'https://github.com/mailinc/mailsuite/imaplib': 'imaplib',
}
def codebaseGenerator(chdict):
return all_repositories[chdict['repository']]
c['codebaseGenerator'] = codebaseGenerator
For any incoming change, the codebase is set to ‘’. This codebase value is sufficient if all changes come from the same repository (or clones). If changes come from different repositories, extra processing will be needed to determine the codebase for the incoming change. This codebase will then be a logical name for the combination of repository and or branch etc.
The codebaseGenerator accepts a change dictionary as produced by the buildbot.db.changes.ChangesConnectorComponent, with a changeid equal to None.
发布评论