Question

Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

3.3.22. Statistics Service

The statistics service (or stats service) is implemented in buildbot.statistics.stats_service. Please see stats-service for more information.

Here is a diagram demonstrating the workings of the stats service:

3.3.22.1. Stats Service

class buildbot.statistics.stats_service.StatsService

Storage backends are responsible for storing any statistics/data sent to them. A storage backend will generally be some sort of a database-server running on a machine.

Note

This machine may be different from the one running BuildMaster.

Data is captured according to the master config file and then is sent to each of the storage backends provided by the master configuration (see stats-service).

Each storage backend has a Python client defined as part of buildbot.statistics.storage_backends to aid in posting data by StatsService.

Currently, only InfluxDB is supported as a storage backend.

class buildbot.statistis.storage_backends.base.StatsStorageBase

Capture classes are used for declaring which data needs to captured and sent to storage backends for storage.

class buildbot.statistics.capture.Capture

This is the abstract base class for all capture classes. Not to be used directly. It’s initialized with the following parameters:

routingKey

(tuple) The routing key to be used by StatsService to register consumers to the MQ layer for the subclass of this class.

callback

The callback registered with the MQ layer for the consumer of a subclass of this class. Each subclass must provide a default callback for this purpose.

_defaultContext(self, msg)

A method for providing default context to the storage backends.

consume(self, routingKey, msg)

This is an abstract method - each subclass of this class should implement its own consume method. If not, then the subclass can’t be instantiated. The consume method, when called (from the mq layer), receives the following arguments:

routingKey

The routing key which was registered to the MQ layer. Same as the routingKey provided to instantiate this class.

msg

The message that was sent by the producer.

_store(self, post_data, series_name, context)

This is an abstract method of this class. It must be implemented by all subclasses of this class. It takes the following arguments:

post_data

(dict) The key-value pair being sent to the storage backend.

series_name

(str) The name of the series to which this data is stored.

context

(dict) Any additional information pertaining to the data being sent.

class buildbot.statistics.capture.CapturePropertyBase

This is a base class for both CaptureProperty and CapturePropertyAllBuilders and abstracts away much of the common functionality between the two classes. It cannot be initialized directly as it contains an abstract method and raises TypeError if tried. It is initialized with the following arguments:

property_name

(str) The name of property needed to be recorded as a statistic. This can be a regular expression if regex=True (see below).

callback=None

The callback function that is used by CaptureProperty.consumer to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this:

default_callback(props, property_name)

It returns property value for property_name. It receives the following arguments:

props

A dictionary of all build properties.

property_name

(str) Name of the build property to return.

regex=False

If this is set to True, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.

consume(self, routingKey, msg)

The consumer for all CaptureProperty classes described below. This method filters out the correct properties as per the configuration file and sends those properties for storage. The subclasses of this method do not need to implement this method as it takes care of all the functionality itself. See Capture for more information.

_builder_name_matches(self, builder_info)

This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the consume method mentioned above. It checks whether a builder is allowed to send properties to the storage backend according to the configuration file. It takes one argument:

builder_info

(dict) The dictionary returned by the data API containing the builder information.

class buildbot.statistics.capture.CaptureProperty

The capture class for capturing build properties. It is available in the configuration as statistics.CaptureProperty.

It takes the following arguments:

builder_name

(str) The name of builder in which the property is recorded.

property_name

(str) The name of property needed to be recorded as a statistic.

callback=None

The callback function that is used by CaptureProperty.consumer to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this (see CapturePropertyBase for more information).

regex=False

If this is set to True, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.

_builder_name_matches(self, builder_info)

See CapturePropertyBase for more information on this method.

class buildbot.statistics.capture.CapturePropertyAllBuilders

The capture class to use for capturing build properties on all builders. It is available in the configuration as statistics.CapturePropertyAllBuilders.

It takes the following arguments:

property_name

(str) The name of property needed to be recorded as a statistic.

callback=None

The callback function that is used by CaptureProperty.consumer to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this (see CapturePropertyBase for more information).

regex=False

If this is set to True, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.

_builder_name_matches(self, builder_info)

See CapturePropertyBase for more information on this method.

class buildbot.statistics.capture.CaptureBuildTimes

A base class for all Capture classes that deal with build times (start/end/duration). Not to be used directly. It’s initialized with:

builder_name

(str) The name of builder whose times are to be recorded.

callback

The callback function that is used by a subclass of this class to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this. Each subclass must provide a default callback that is used in initialization of this class should the user not provide a callback.

consume(self, routingKey, msg)

The consumer for all subclasses of this class. See Capture for more information.

Note

This consumer requires all subclasses to implement:

self._time_type

A string used as a key in post_data sent to storage services.

self._retValParams(msg)

A method that takes in the msg this consumer gets and returns a list of arguments for the capture callback.

_retValParams(self, msg)

This is an abstract method which needs to be implemented by subclasses. This method needs to return a list of parameters that will be passed to the callback function. See individual build CaptureBuild* classes for more information.

_err_msg(self, build_data, builder_name)

A helper method that returns an error message for the consume method.

_builder_name_matches(self, builder_info)

This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the consume method mentioned above. It checks whether a builder is allowed to send build times to the storage backend according to the configuration file. It takes one argument:

builder_info

(dict) The dictionary returned by the data API containing the builder information.

class buildbot.statistics.capture.CaptureBuildStartTime

A capture class for capturing build start times. It takes the following arguments:

builder_name

(str) The name of builder whose times are to be recorded.

callback=None

The callback function for this class. See CaptureBuildTimes for more information.

The default callback:

default_callback(start_time)

It returns the start time in ISO format. It takes one argument:

start_time

A python datetime object that denotes the build start time.

_retValParams(self, msg)

Returns a list containing one Python datetime object (start time) from msg dictionary.

_builder_name_matches(self, builder_info)

See CaptureBuildTimes for more information on this method.

class buildbot.statistics.capture.CaptureBuildStartTimeAllBuilders

A capture class for capturing build start times from all builders. It is a subclass of CaptureBuildStartTime. It takes the following arguments:

callback=None

The callback function for this class. See CaptureBuildTimes for more information.

The default callback:

See CaptureBuildStartTime.__init__ for the definition.

_builder_name_matches(self, builder_info)

See CaptureBuildTimes for more information on this method.

class buildbot.statistics.capture.CaptureBuildEndTime

A capture class for capturing build end times. Takes the following arguments:

builder_name

(str) The name of builder whose times are to be recorded.

callback=None

The callback function for this class. See CaptureBuildTimes for more information.

The default callback:

default_callback(end_time)

It returns the end time in ISO format. It takes one argument:

end_time

A python datetime object that denotes the build end time.

_retValParams(self, msg)

Returns a list containing two Python datetime object (start time and end time) from msg dictionary.

_builder_name_matches(self, builder_info)

See CaptureBuildTimes for more information on this method.

class buildbot.statistics.capture.CaptureBuildEndTimeAllBuilders

A capture class for capturing build end times from all builders. It is a subclass of CaptureBuildEndTime. It takes the following arguments:

callback=None

The callback function for this class. See CaptureBuildTimes for more information.

The default callback:

See CaptureBuildEndTime.__init__ for the definition.

_builder_name_matches(self, builder_info)

See CaptureBuildTimes for more information on this method.

class buildbot.statistics.capture.CaptureBuildDuration

A capture class for capturing build duration. Takes the following arguments:

builder_name

(str) The name of builder whose times are to be recorded.

report_in='seconds'

Can be one of three: 'seconds', 'minutes', or 'hours'. This is the units in which the build time will be reported.

callback=None

The callback function for this class. See CaptureBuildTimes for more information.

The default callback:

default_callback(start_time, end_time)

It returns the duration of the build as per the report_in argument. It receives the following arguments:

start_time

A python datetime object that denotes the build start time.

end_time

A python datetime object that denotes the build end time.

_retValParams(self, msg)

Returns a list containing one Python datetime object (end time) from msg dictionary.

_builder_name_matches(self, builder_info)

See CaptureBuildTimes for more information on this method.

class buildbot.statistics.capture.CaptureBuildDurationAllBuilders

A capture class for capturing build durations from all builders. It is a subclass of CaptureBuildDuration. It takes the following arguments:

callback=None

The callback function for this class. See CaptureBuildTimes for more.

The default callback:

See CaptureBuildDuration.__init__ for the definition.

_builder_name_matches(self, builder_info)

See CaptureBuildTimes for more information on this method.

class buildbot.statistics.capture.CaptureDataBase

This is a base class for both CaptureData and CaptureDataAllBuilders and abstracts away much of the common functionality between the two classes. Cannot be initialized directly as it contains an abstract method and raises TypeError if tried. It is initialized with the following arguments:

data_name

(str) The name of data to be captured. Same as in yieldMetricsValue.

callback=None

The callback function for this class.

The default callback:

The default callback takes a value x and return it without changing. As such, x itself acts as the post_data sent to the storage backends.

consume(self, routingKey, msg)

The consumer for this class. See Capture for more.

_builder_name_matches(self, builder_info)

This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the consume method mentioned above. It checks whether a builder is allowed to send properties to the storage backend according to the configuration file. It takes one argument:

builder_info

(dict) The dictionary returned by the data API containing the builder information.

class buildbot.statistics.capture.CaptureData

A capture class for capturing arbitrary data that is not stored as build-data. See yieldMetricsValue for more. Takes the following arguments for initialization:

data_name

(str) The name of data to be captured. Same as in yieldMetricsValue.

builder_name

(str) The name of the builder on which the data is captured.

callback=None

The callback function for this class.

The default callback:

See CaptureDataBase of definition.

_builder_name_matches(self, builder_info)

See CaptureDataBase for more information on this method.

class buildbot.statistics.capture.CaptureDataAllBuilders

A capture class to capture arbitrary data on all builders. See yieldMetricsValue for more. It takes the following arguments:

data_name

(str) The name of data to be captured. Same as in yieldMetricsValue.

callback=None

The callback function for this class.

_builder_name_matches(self, builder_info)

See CaptureDataBase for more information on this method.