Question

Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

3.6. Data API

The data API is an interface against which various internal and external components can be written. It is a lower-level interface compared to the REST API that exposes more functionality. It combines access to stored state and messages, ensuring consistency between them. The callers can receive a dump of the current state plus changes to that state, without missing or duplicating messages.

3.6.1. Sections

The data API is divided into four sections:

• getters - fetching data from the db API

• subscriptions - subscribing to messages from the mq layer

• control - allows state to be changed in specific ways by sending appropriate messages (e.g., stopping a build)

• updates - direct updates to state appropriate messages.

The getters and subscriptions are exposed everywhere. Access to the control section must be authenticated at higher levels as the data layer does no authentication. The updates section is for use only by the process layer.

The interfaces for all sections, but the updates section, are intended to be language-agnostic. That is, they should be callable from JavaScript via HTTP, or via some other interface added to Buildbot after the fact.

3.6.1.1. Getters

The getters section can get either a single resource, or a list of resources. Getting a single resource requires a resource identifier (a tuple of strings) and a set of options to support automatic expansion of links to other resources (thus saving round-trips). Lists are requested with a partial resource identifier (a tuple of strings) and an optional set of filter options. In some cases, certain filters are implicit in the path, e.g., the list of buildsteps for a particular build.

3.6.1.2. Subscriptions

Message subscriptions can be made to anything that can be listed or gotten from the getters section, using the same resource identifiers. Options and explicit filters are not supported here. A message contains only the most basic information about a resource and a list of subscription results for every new resource of the desired type. Implicit filters are supported.

3.6.1.3. Control

The control section defines a set of actions that cause Buildbot to behave in a certain way, e.g., rebuilding a build or shutting down a worker. Actions correspond to a particular resource, although sometimes that resource is the root resource (an empty tuple).

3.6.1.4. Updates

The updates section defines a free-form set of methods that Buildbot’s process implementation calls to update data. Most update methods both modify state via the db API and send a message via the mq API. Some are simple wrappers for these APIs, while others contain more complex logic, e.g., building a source stamp set for a collection of changes. This section is the proper place to put common functionality, e.g., rebuilding builds or assembling buildsets.

3.6.2. Concrete Interfaces

3.6.2.1. Python Interface

Within the buildmaster process, the root of the data API is available at self.master.data, which is a

The updates section is available at self.master.data.updates, and contains a number of ad-hoc methods needed by the process modules.

Note

The update methods are implemented in resource type classes, but through some initialization-time magic, all appear as attributes of self.master.data.updates.

The update methods are found in the resource type pages.

Exceptions

exception buildbot.data.exceptions.DataException

The HTTP interface is implemented by the buildbot.www package, as configured by the user. Part of that configuration is a base URL, which is considered a prefix for all paths mentioned here.

See Base web application for more information.

3.6.3. Extending the Data API

The data API may be extended in various ways: adding new endpoints, new fields to resource types, new update methods, or entirely new resource types. In any case, you should only extend the API if you plan to submit the extensions to be merged into Buildbot itself. Private API extensions are strongly discouraged.

3.6.3.1. Adding Resource Types

You’ll need to use both plural and singular forms of the resource type; in this example, we’ll use ‘pub’ and ‘pubs’. You can also examine an existing file, like master/buildbot/data/changes.py, to see when to use which form.

In master/buildbot/data/pubs.py, create a subclass of

Each resource path is implemented as an

Message types are defined in master/buildbot/test/util/validation.py, via the message module-level value. This is a dictionary of MessageValidator objects, one for each message type. The message type is determined from the first atom of its routing key. The events dictionary lists the possible last atoms of the routing key. It should be identical to the attribute of the ResourceType with the same name.

3.6.3.4. Adding Update Methods

Update methods are for use by the Buildbot process code, and as such are generally designed to suit the needs of that code. They generally encapsulate logic common to multiple users (e.g., creating buildsets), and they finish by performing modifications in the database and sending a corresponding message. In general, Buildbot does not depend on timing of either the database or message broker, so the order in which these operations are initiated is not important.

Update methods are considered part of Buildbot’s user-visible interface, and as such, incompatible changes should be avoided wherever possible. Instead, either add a new method (and potentially re-implement existing methods in terms of the new method) or add new, optional parameters to an existing method. If an incompatible change is unavoidable, it should be described clearly in the release notes.

Update methods are implemented as methods of

The details of the fields of a resource type are rigorously enforced at several points in the Buildbot tests. The enforcement is performed by the

class buildbot.data.types.Integer

class buildbot.data.types.NoneOk(nestedType)

class buildbot.data.types.Entity(name)

The data API enforces a strong and well-defined model on Buildbot’s data. This model is influenced by REST, in the sense that it defines resources, representations for resources, and identifiers for resources. For each resource type, the API specifies:

• the attributes of the resource and their types (e.g., changes have a string specifying their project)

• the format of links to other resources (e.g., buildsets to sourcestamp sets)

• the paths relating to the resource type

• the format of routing keys for messages relating to the resource type

• the events that can occur on that resource (e.g., a buildrequest can be claimed)

• options and filters for getting resources

Some resource type attributes only appear in certain formats, as noted in the documentation for the resource types. In general, messages do not include any optional attributes, nor links.

Paths are given here separated by slashes, with key names prefixed by : and described below. Similarly, message routing keys given here are separated by dots, with key names prefixed by $. The translation to tuples and other formats should be obvious.

All strings in the data model are unicode strings.