Question

Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

3.9. Messaging and Queues

Buildbot uses a message-queueing structure to handle asynchronous notifications in a distributed fashion. This avoids, for the most part, the need for each master to poll the database, allowing masters to react to events as they happen.

3.9.1. Overview

Buildbot is structured as a hybrid state- and event-based application, which will probably offend adherents of either pattern. In particular, the most current state is stored in the Database, while any changes to the state are announced in the form of a message. The content of the messages is sufficient to reconstruct the updated state, allowing external processes to represent “live” state without polling the database.

This split nature immediately brings to light the problem of synchronizing the two interfaces. Queueing systems can introduce queueing delays as messages propagate. Likewise, database systems may introduce a delay between committed modifications and the modified data appearing in queries; for example, with MySQL master/slave replication, there can be several seconds’ delay before a slave is updated.

Buildbot’s MQ connector simply relays messages, and makes no attempt to coordinate the timing of those messages with the corresponding database updates. It is up to higher layers to apply such coordination.

3.9.2. Connector API

All access to the queueing infrastructure is mediated by an MQ connector. The connector’s API is defined below. The connector itself is always available as master.mq, where master is the current BuildMaster instance.

The connector API is quite simple. It is loosely based on AMQP, although simplified because there is only one exchange (see

Several concrete implementations of the MQ connector exist. The simplest is intended for cases where only one master exists, similar to the SQLite database support. The remainder use various existing queueing applications to support distributed communications.

Simple

class buildbot.mq.simple.SimpleMQ

class buildbot.mq.wamp.WampMQ

Buildbot uses a particularly simple architecture: in AMQP terms, all messages are sent to a single topic exchange, and consumers define anonymous queues bound to that exchange.

In future versions of Buildbot, some components (e.g., schedulers) may use durable queues to ensure that messages are not lost when one or more masters are disconnected.

3.9.4. Message Schema

This section describes the general structure messages. The specific routing keys and content of each message are described in the relevant sub-sections of Data API.

3.9.4.1. Routing Keys

Routing keys are a sequence of strings, usually written with dot separators. Routing keys are represented with variables when one or more of the words in the key are defined by the content of the message. For example, buildset.$bsid describes routing keys such as buildset.1984, where 1984 is the ID of the buildset described by the message body. Internally, keys are represented as tuples of strings.

3.9.4.2. Body Format

Message bodies are encoded in JSON. The top level of each message is an object (a dictionary).

Most simple Python types - strings, numbers, lists, and dictionaries - are mapped directly to the corresponding JSON types. Timestamps are represented as seconds since the UNIX epoch in message bodies.

3.9.4.3. Cautions

Message ordering is generally maintained by the backend implementations, but this should not be depended on. That is, messages originating from the same master are usually delivered to consumers in the order they were produced. Thus, for example, a consumer can expect to see a build request claimed before it is completed. That said, consumers should be resilient to messages delivered out of order, at the very least by scheduling a “reload” from state stored in the database when messages arrive in an invalid order.

Unit tests should be used to ensure this resiliency.

Some related messages are sent at approximately the same time. Due to the non-blocking nature of message delivery, consumers should not assume that subsequent messages in a sequence remain queued. For example, upon receipt of a buildset.$bsid.new message, it is already too late to try to subscribe to the associated build requests messages, as they may already have been consumed.

3.9.4.4. Schema Changes

Future versions of Buildbot may add keys to messages, or add new messages. Consumers should expect unknown keys and, if using wildcard topics, unknown messages.