The
ChangeSources will, in general, automatically provide the proper repository attribute for any changes they produce. For systems which operate on URL-like specifiers, this is a repository URL. Other ChangeSources adapt the concept as necessary.
Many ChangeSources allow you to specify a project, as well. This attribute is useful when building from several distinct codebases in the same buildmaster: the project string can serve to differentiate the different codebases. Schedulers can filter on project, so you can configure different builders to run for each project.
Many projects publish information about changes to their source tree by sending an email message out to a mailing list, frequently named PROJECT-commits or PROJECT-changes. Each message usually contains a description of the change (who made the change, which files were affected) and sometimes a copy of the diff. Humans can subscribe to this list to stay informed about what’s happening to the source tree.
Buildbot can also subscribe to a -commits mailing list, and can trigger builds in response to Changes that it hears about. The buildmaster admin needs to arrange for these email messages to arrive in a place where the buildmaster can find them, and configure the buildmaster to parse the messages correctly. Once that is in place, the email parser will create Change objects and deliver them to the schedulers (see Schedulers) just like any other ChangeSource.
There are two components to setting up an email-based ChangeSource. The first is to route the email messages to the buildmaster, which is done by dropping them into a maildir. The second is to actually parse the messages, which is highly dependent upon the tool that was used to create them. Each VC system has a collection of favorite change-emailing tools with a slightly different format and its own parsing function. Buildbot has a separate ChangeSource variant for each of these parsing functions.
Once you’ve chosen a maildir location and a parsing function, create the change source and put it in
The recommended way to install Buildbot is to create a dedicated account for the buildmaster. If you do this, the account will probably have a distinct email address (perhaps buildmaster@example.org). Then just arrange for this account’s email to be delivered to a suitable maildir (described in the next section).
If Buildbot does not have its own account, extension addresses can be used to distinguish between emails intended for the buildmaster and emails intended for the rest of the account. In most modern MTAs, the e.g. foo@example.org account has control over every email address at example.org which begins with “foo”, such that emails addressed to account-foo@example.org can be delivered to a different destination than account-bar@example.org. qmail does this by using separate .qmail files for the two destinations (.qmail-foo and .qmail-bar, with .qmail controlling the base address and .qmail-default controlling all other extensions). Other MTAs have similar mechanisms.
Thus you can assign an extension address like foo-buildmaster@example.org to the buildmaster and retain foo@example.org for your own use.
A maildir is a simple directory structure originally developed for qmail that allows safe atomic update without locking. Create a base directory with three subdirectories: new, tmp, and cur. When messages arrive, they are put into a uniquely-named file (using pids, timestamps, and random numbers) in tmp. When the file is complete, it is atomically renamed into new. Eventually the buildmaster notices the file in new, reads and parses the contents, then moves it into cur. A cronjob can be used to delete files in cur at leisure.
Maildirs are frequently created with the maildirmake tool, but a simple mkdir -p ~/MAILDIR/cur,new,tmp is pretty much equivalent.
Many modern MTAs can deliver directly to maildirs. The usual .forward or .procmailrc syntax is to name the base directory with a trailing slash, so something like ~/MAILDIR/. qmail and postfix are maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail Delivery Agent).
Here is an example procmail config, located in ~/.procmailrc:
# .procmailrc
# routes incoming mail to appropriate mailboxes
PATH=/usr/bin:/usr/local/bin
MAILDIR=$HOME/Mail
LOGFILE=.procmail_log
SHELL=/bin/sh
:0
*
new
If procmail is not setup on a system wide basis, then the following one-line .forward file will invoke it.
!/usr/bin/procmail
For MTAs which cannot put files into maildirs directly, the safecat tool can be executed from a .forward file to accomplish the same thing.
The Buildmaster uses the linux DNotify facility to receive immediate notification when the maildir’s new directory has changed. When this facility is not available, it polls the directory for new messages, every 10 seconds by default.
The second component to setting up an email-based ChangeSource is to parse the actual notices. This is highly dependent upon the VC system and commit script in use.
A couple of common tools used to create these change emails, along with the Buildbot tools to parse them, are:
- CVS
- Buildbot CVS MailNotifier
- class buildbot.changes.mail.CVSMaildirSource
CVS must be configured to invoke the buildbot_cvs_mail.py script when files are checked in. This is done via the CVS loginfo configuration file.
To update this, first do:
cvs checkout CVSROOT
cd to the CVSROOT directory and edit the file loginfo, adding a line like:
SomeModule /cvsroot/CVSROOT/buildbot_cvs_mail.py --cvsroot :ext:example.com:/cvsroot -e buildbot -P SomeModule %@{sVv@}
Note
For cvs version 1.12.x, the --path %p option is required. Version 1.11.x and 1.12.x report the directory path differently.
The above example you put the buildbot_cvs_mail.py script under /cvsroot/CVSROOT. It can be anywhere. Run the script with --help to see all the options. At the very least, the options -e (email) and -P (project) should be specified. The line must end with %{sVv}. This is expanded to the files that were modified.
Additional entries can be added to support more modules.
See buildbot_cvs_mail.py --help for more information on the available options.
- class buildbot.changes.mail.SVNCommitEmailMaildirSource
- class buildbot.changes.mail.BzrLaunchpadEmailMaildirSource
- class buildbot.changes.pb.PBChangeSource
Bzr is also written in Python, and the Bzr hook depends on Twisted to send the changes.
To install, put master/contrib/bzr_buildbot.py in one of your plugins locations a bzr plugins directory (e.g., ~/.bazaar/plugins). Then, in one of your bazaar conf files (e.g., ~/.bazaar/locations.conf), set the location you want to connect with Buildbot with these keys:
buildbot_on one of ‘commit’, ‘push, or ‘change’. Turns the plugin on to report changes via commit, changes via push, or any changes to the trunk. ‘change’ is recommended.
buildbot_server (required to send to a Buildbot master) the URL of the Buildbot master to which you will connect (as of this writing, the same server and port to which workers connect).
buildbot_port (optional, defaults to 9989) the port of the Buildbot master to which you will connect (as of this writing, the same server and port to which workers connect)
buildbot_pqm (optional, defaults to not pqm) Normally, the user that commits the revision is the user that is responsible for the change. When run in a pqm (Patch Queue Manager, see https://launchpad.net/pqm) environment, the user that commits is the Patch Queue Manager, and the user that committed the parent revision is responsible for the change. To turn on the pqm mode, set this value to any of (case-insensitive) “Yes”, “Y”, “True”, or “T”.
buildbot_dry_run (optional, defaults to not a dry run) Normally, the post-commit hook will attempt to communicate with the configured Buildbot server and port. If this parameter is included and any of (case-insensitive) “Yes”, “Y”, “True”, or “T”, then the hook will simply print what it would have sent, but not attempt to contact the Buildbot master.
buildbot_send_branch_name (optional, defaults to not sending the branch name) If your Buildbot’s bzr source build step uses a repourl, do not turn this on. If your buildbot’s bzr build step uses a baseURL, then you may set this value to any of (case-insensitive) “Yes”, “Y”, “True”, or “T” to have the Buildbot master append the branch name to the baseURL.
Note
The bzr smart server (as of version 2.2.2) doesn’t know how to resolve bzr:// urls into absolute paths so any paths in locations.conf won’t match, hence no change notifications will be sent to Buildbot. Setting configuration parameters globally or in-branch might still work. When Buildbot no longer has a hardcoded password, it will be a configuration option here as well.
Here’s a simple example that you might have in your ~/.bazaar/locations.conf.
[chroot-*:///var/local/myrepo/mybranch]
buildbot_on = change
buildbot_server = localhost
The
This configuration uses the P4PORT, P4USER, and P4PASSWD specified in the buildmaster’s environment. It watches a project in which the branch name is simply the next path component, and the file is all path components after.
from buildbot.plugins import changes
s = changes.P4Source(p4base='//depot/project/',
split_file=lambda branchfile: branchfile.split('/',1))
c['change_source'] = s
Similar to the previous example but also resolves the branch and revision into a valid revlink.
from buildbot.plugins import changes
s = changes.P4Source(
p4base='//depot/project/',
split_file=lambda branchfile: branchfile.split('/',1))
revlink=lambda branch, revision: 'http://p4web:8080/@md=d&@/{}?ac=10'.format(revision)
c['change_source'] = s
- class buildbot.changes.svnpoller.SVNPoller
If you cannot insert a Bzr hook in the server, you can use the
If you cannot take advantage of post-receive hooks as provided by master/contrib/git_buildbot.py for example, then you can use the
The
- class buildbot.changes.github.GitHubPullrequestPoller
- class buildbot.changes.bitbucket.BitbucketPullrequestPoller
- class buildbot.changes.gerritchangesource.GerritChangeSource
- class buildbot.changes.gerritchangesource.GerritEventLogPoller
- class buildbot.changes.gerritchangesource.GerritChangeFilter
Buildbot already provides a web frontend, and that frontend can easily be used to receive HTTP push notifications of commits from services like GitHub. See Change Hooks for more information.
2.5.4. Changes
- class buildbot.changes.changes.Change
Each Change has a who attribute, which specifies which developer is responsible for the change. This is a string which comes from a namespace controlled by the VC repository. Frequently this means it is a username on the host which runs the repository, but not all VC systems require this. Each StatusNotifier will map the who attribute into something appropriate for their particular means of communication: an email address, an IRC handle, etc.
This who attribute is also parsed and stored into Buildbot’s database (see User Objects). Currently, only who attributes in Changes from git repositories are translated into user objects, but in the future all incoming Changes will have their who parsed and stored.
2.5.4.2. Files
It also has a list of files, which are just the tree-relative filenames of any files that were added, deleted, or modified for this Change. These filenames are checked by the fileIsImportant function of a scheduler to decide whether it should trigger a new build or not. For example, the scheduler could use the following function to only run a build if a C file was checked in:
def has_C_files(change):
for name in change.files:
if name.endswith(".c"):
return True
return False
Certain BuildSteps can also use the list of changed files to run a more targeted series of tests, e.g. the python_twisted.Trial step can run just the unit tests that provide coverage for the modified .py files instead of running the full test suite.
2.5.4.3. Comments
The Change also has a comments attribute, which is a string containing any checkin comments.
2.5.4.4. Project
The project attribute of a change or source stamp describes the project to which it corresponds, as a short human-readable string. This is useful in cases where multiple independent projects are built on the same buildmaster. In such cases, it can be used to control which builds are scheduled for a given commit, and to limit status displays to only one project.
2.5.4.5. Repository
This attribute specifies the repository in which this change occurred. In the case of DVCS’s, this information may be required to check out the committed source code. However, using the repository from a change has security risks: if Buildbot is configured to blindly trust this information, then it may easily be tricked into building arbitrary source code, potentially compromising the workers and the integrity of subsequent builds.
2.5.4.6. Codebase
This attribute specifies the codebase to which this change was made. As described in source stamps section, multiple repositories may contain the same codebase. A change’s codebase is usually determined by the codebaseGenerator configuration. By default the codebase is ‘’; this value is used automatically for single-codebase configurations.
2.5.4.7. Revision
Each Change can have a revision attribute, which describes how to get a tree with a specific state: a tree which includes this Change (and all that came before it) but none that come after it. If this information is unavailable, the revision attribute will be None. These revisions are provided by the ChangeSource.
Revisions are always strings.
- CVS
revision is the seconds since the epoch as an integer.
- SVN
revision is the revision number
- Darcs
revision is a large string, the output of darcs changes --context
- Mercurial
revision is a short string (a hash ID), the output of hg identify
- P4
revision is the transaction number
- Git
revision is a short string (a SHA1 hash), the output of e.g. git rev-parse
2.5.4.8. Branches
The Change might also have a branch attribute. This indicates that all of the Change’s files are in the same named branch. The schedulers get to decide whether the branch should be built or not.
For VC systems like CVS, Git, Mercurial and Monotone the branch name is unrelated to the filename. (That is, the branch name and the filename inhabit unrelated namespaces.) For SVN, branches are expressed as subdirectories of the repository, so the file’s repourl is a combination of some base URL, the branch name, and the filename within the branch. (In a sense, the branch name and the filename inhabit the same namespace.) Darcs branches are subdirectories of a base URL just like SVN.
- CVS
branch=’warner-newfeature’, files=[‘src/foo.c’]
- SVN
branch=’branches/warner-newfeature’, files=[‘src/foo.c’]
- Darcs
branch=’warner-newfeature’, files=[‘src/foo.c’]
- Mercurial
branch=’warner-newfeature’, files=[‘src/foo.c’]
- Git
branch=’warner-newfeature’, files=[‘src/foo.c’]
- Monotone
branch=’warner-newfeature’, files=[‘src/foo.c’]
2.5.4.9. Change Properties
A Change may have one or more properties attached to it, usually specified through the Force Build form or sendchange. Properties are discussed in detail in the Build Properties section.
发布评论