Question

Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

2.5.5. Schedulers

• The

  In a multi-master configuration, schedulers with the same name can be configured on multiple masters. Only one instance of the scheduler will be active. If that instance becomes inactive, due to its master being shut down or failing, then another instance will become active after a short delay. This provides resiliency in scheduler configurations, so that schedulers are not a single point of failure in a Buildbot infrastructure.

  The Data API and web UI display the master on which each scheduler is running.

  There is currently no mechanism to control which master’s scheduler instance becomes active. The behavior is nondeterministic, based on the timing of polling by inactive schedulers. The failover is non-revertive.

  A quick scheduler might exist to give immediate feedback to developers, hoping to catch obvious problems in the code that can be detected quickly. These typically do not run the full test suite, nor do they run on a wide variety of platforms. They also usually do a VC update rather than performing a brand-new checkout each time.

  A separate full scheduler might run more comprehensive tests, to catch more subtle problems. It might be configured to run after the quick scheduler, to give developers time to commit fixes to bugs caught by the quick scheduler before running the comprehensive tests. This scheduler would also feed multiple Builders.

  Many schedulers can be configured to wait a while after seeing a source-code change - this is the tree stable timer. The timer allows multiple commits to be “batched” together. This is particularly useful in distributed version control systems, where a developer may push a long sequence of changes all at once. To save resources, it’s often desirable only to test the most recent change.

  Schedulers can also filter out the changes they are interested in, based on a number of criteria. For example, a scheduler that only builds documentation might skip any changes that do not affect the documentation. Schedulers can also filter on the branch to which a commit was made.

  Periodic builds (those which are run every N seconds rather than after new Changes arrive) are triggered by a special

  This is the original and still most popular scheduler class. It follows exactly one branch, and starts a configurable tree-stable-timer after each change on that branch. When the timer expires, it starts a build on some set of Builders. This scheduler accepts a fileIsImportant function which can be used to ignore some Changes if they do not affect any important files.

  If treeStableTimer is not set, then this scheduler starts a build for every Change that matches its change_filter and satisfies fileIsImportant. If treeStableTimer is set, then a build is triggered for each set of Changes that arrive in intervals shorter than the configured time and match the filters.

  Note

  The behavior of this scheduler is undefined, if treeStableTimer is set, and changes from multiple branches, repositories or codebases are accepted by the filter.

  Note

  The codebases argument will filter out codebases not specified there, but won’t filter based on the branches specified there.

  The arguments to this scheduler are:

  name

  See

  This scheduler uses a tree-stable-timer like the default one, but uses a separate timer for each branch.

  If treeStableTimer is not set, then this scheduler is indistinguishable from

  It is common to wind up with one kind of build which should only be performed if the same source code was successfully handled by some other kind of build first. An example might be a packaging step: you might only want to produce .deb or RPM packages from a tree that was known to compile successfully and pass all unit tests. You could put the packaging step in the same Build as the compile and testing steps, but there might be other reasons to not do this (in particular you might have several Builders worth of compiles/tests, but only wish to do the packaging once). Another example is if you want to skip the full builds after a failing quick build of the same source code. Or, if one Build creates a product (like a compiled library) that is used by some other Builder, you’d want to make sure the consuming Build is run after the producing one.

  You can use dependencies to express this relationship to Buildbot. There is a special kind of scheduler named

  This simple scheduler just triggers a build every N seconds.

  The arguments to this scheduler are:

  name

  See

  This is highly configurable periodic build scheduler, which triggers a build at particular times of day, week, month, or year. The configuration syntax is very similar to the well-known crontab format, in which you provide values for minute, hour, day, and month (some of which can be wildcards), and a build is triggered whenever the current time matches the given constraints. This can run a build every night, every morning, every weekend, alternate Thursdays, on your boss’s birthday, etc.

  Pass some subset of minute, hour, dayOfMonth, month, and dayOfWeek; each may be a single number or a list of valid values. The builds will be triggered whenever the current time matches these values. Wildcards are represented by a ‘*’ string. All fields default to a wildcard except ‘minute’, so with no fields, this defaults to a build every hour, on the hour. The full list of parameters is:

  name

  See

  This scheduler allows developers to use the buildbot try command to trigger builds of code they have not yet committed. See try for complete details.

  Two implementations are available:

  The

  class buildbot.schedulers.timed.NightlyTriggerable

  The

  The force scheduler uses the web interface’s authorization framework to determine which user has the right to force which build. Here is an example of code on how you can define which user has which right:

  user_mapping = {
      re.compile("project1-builder"): ["project1-maintainer", "john"] ,
      re.compile("project2-builder"): ["project2-maintainer", "jack"],
      re.compile(".*"): ["root"]
  }
  def force_auth(user,  status):
      global user_mapping
      for r,users in user_mapping.items():
          if r.match(status.name):
              if user in users:
                      return True
      return False
  
  # use authz_cfg in your WebStatus setup
  authz_cfg=authz.Authz(
      auth=my_auth,
      forceBuild = force_auth,
  )
  

  ForceScheduler Parameters

  Most of the arguments to

  NestedParameter(name="options", label="Build options", layout="vertical", fields=[...]),
  

  This parameter type is a special parameter which contains other parameters. This can be used to group a set of parameters together, and define the layout of your form. You can recursively include NestedParameter into NestedParameter, to build very complex UIs.

  It adds the following arguments:

  layout (optional, default is “vertical”)

  The layout defines how the fields are placed in the form.

  The layouts implemented in the standard web application are:

  • simple: fields are displayed one by one without alignment.

    They take the horizontal space that they need.

  • vertical: all fields are displayed vertically, aligned in columns (as per the column attribute of the NestedParameter)

  • tabs: each field gets its own tab.

    This can be used to declare complex build forms which won’t fit into one screen. The children fields are usually other NestedParameters with vertical layout.

  columns (optional, accepted values are 1, 2, 3, 4)

  The number of columns to use for a vertical layout. If omitted, it is set to 1 unless there are more than 3 visible child fields in which case it is set to 2.

  FixedParameter

  FixedParameter(name="branch", default="trunk"),
  

  This parameter type will not be shown on the web form and always generates a property with its default value.

  StringParameter

  StringParameter(name="pull_url",
      label="optionally give a public Git pull url:",
      default="", size=80)
  

  This parameter type will show a single-line text-entry box, and allow the user to enter an arbitrary string. It adds the following arguments:

  regex (optional)

  A string that will be compiled as a regex and used to validate the input of this parameter.

  size (optional; default is 10)

  The width of the input field (in characters).

  TextParameter

  TextParameter(name="comments",
      label="comments to be displayed to the user of the built binary",
      default="This is a development build", cols=60, rows=5)
  

  This parameter type is similar to StringParameter, except that it is represented in the HTML form as a textarea, allowing multi-line input. It adds the StringParameter arguments and the following ones:

  cols (optional; default is 80)

  The number of columns the textarea will have.

  rows (optional; default is 20)

  The number of rows the textarea will have.

  This class could be subclassed to have more customization, e.g.

  • developer could send a list of Git branches to pull from

  • developer could send a list of Gerrit changes to cherry-pick,

  • developer could send a shell script to amend the build.

  Beware of security issues anyway.

  IntParameter

  IntParameter(name="debug_level",
      label="debug level (1-10)", default=2)
  

  This parameter type accepts an integer value using a text-entry box.

  BooleanParameter

  BooleanParameter(name="force_build_clean",
      label="force a make clean", default=False)
  

  This type represents a boolean value. It will be presented as a checkbox.

  UserNameParameter

  UserNameParameter(label="your name:", size=80)
  

  This parameter type accepts a username. If authentication is active, it will use the authenticated user instead of displaying a text-entry box.

  size (optional; default is 10)

  The width of the input field (in characters).

  need_email (optional; default is True)

  If true, requires a full email address rather than arbitrary text.

  ChoiceStringParameter

  ChoiceStringParameter(name="branch",
      choices=["main","devel"], default="main")
  

  This parameter type lets the user choose between several choices (e.g. the list of branches you are supporting, or the test campaign to run). If multiple is false, then its result is a string with one of the choices. If multiple is true, then the result is a list of strings from the choices.

  Note that for some use cases, the choices need to be generated dynamically. This can be done via subclassing and overriding the ‘getChoices’ member function. An example of this is provided by the source for the InheritBuildParameter class.

  Its arguments, in addition to the common options, are:

  choices

  The list of available choices.

  strict (optional; default is True)

  If true, verify that the user’s input is from the list. Note that this only affects the validation of the form request; even if this argument is False, there is no HTML form component available to enter an arbitrary value.

  multiple

  If true, then the user may select multiple choices.

  Example:

  ChoiceStringParameter(name="forced_tests",
                        label="smoke test campaign to run",
                        default=default_tests,
                        multiple=True,
                        strict=True,
                        choices=["test_builder1", "test_builder2",
                                 "test_builder3"])
  
  # .. and later base the schedulers to trigger off this property:
  
  # triggers the tests depending on the property forced_test
  builder1.factory.addStep(Trigger(name="Trigger tests",
                                  schedulerNames=Property("forced_tests")))
  

  Example of scheduler allowing to choose which worker to run on:

  worker_list = ["worker1", "worker2", "worker3"]
  ChoiceStringParameter(name="worker",
                        label="worker to run the build on",
                        default="*",
                        multiple=False,
                        strict=True,
                        choices=worker_list)
  
  # .. and in nextWorker, use this property:
  def nextWorker(bldr, workers, buildrequest):
      forced_worker = buildrequest.properties.getProperty("worker", "*")
      if forced_worker == "*":
          return random.choice(workers) if workers else None
      for w in workers:
          if w.worker.workername == forced_worker:
              return w
      return None  # worker not yet available
  
  c['builders'] = [
    BuilderConfig(name='mybuild', factory=f, nextWorker=nextWorker,
          workernames=worker_list),
  ]
  

  CodebaseParameter

  CodebaseParameter(codebase="myrepo")
  

  This is a parameter group to specify a sourcestamp for a given codebase.

  codebase

  The name of the codebase.

  branch (optional; default is StringParameter)

  A

  This parameter allows the user to upload a file to a build. The user can either write some text to a text area, or select a file from the browser. Note that the file is then stored inside a property, so a maxsize of 10 megabytes has been set. You can still override that maxsize if you wish.

  PatchParameter

  This parameter allows the user to specify a patch to be applied at the source step. The patch is stored within the sourcestamp, and associated to a codebase. That is why

  Note

  InheritBuildParameter is not yet ported to data API, and cannot be used with buildbot nine yet (bug #3521).

  This is a special parameter for inheriting force build properties from another build. The user is presented with a list of compatible builds from which to choose, and all forced-build parameters from the selected build are copied into the new build. The new parameter is:

  compatible_builds

  A function to find compatible builds in the build history. This function is given the master instance as first argument, and the current builder name as second argument, or None when forcing all builds.

  Example:

  @defer.inlineCallbacks
  def get_compatible_builds(master, builder):
      if builder is None: # this is the case for force_build_all
          return ["cannot generate build list here"]
      # find all successful builds in builder1 and builder2
      builds = []
      for builder in ["builder1", "builder2"]:
          # get 40 last builds for the builder
          build_dicts = yield master.data.get(('builders', builder, 'builds'),
                                              order=['-buildid'], limit=40)
          for build_dict in build_dicts:
              if build_dict['results'] != SUCCESS:
                  continue
              builds.append(builder + "/" + str(build_dict['number']))
      return builds
  
  # ...
  
  sched = Scheduler(...,
      properties=[
          InheritBuildParameter(
              name="inherit",
              label="promote a build for merge",
              compatible_builds=get_compatible_builds,
              required = True),
              ])
  

  WorkerChoiceParameter

  Note

  WorkerChoiceParameter is not yet ported to data API, and cannot be used with buildbot nine yet (bug #3521).

  This parameter allows a scheduler to require that a build is assigned to the chosen worker. The choice is assigned to the workername property for the build. The enforceChosenWorker functor must be assigned to the canStartBuild parameter for the Builder.

  Example:

  from buildbot.plugins import util
  
  # schedulers:
  ForceScheduler(
      # ...
      properties=[
          WorkerChoiceParameter(),
      ]
  )
  
  # builders:
  BuilderConfig(
      # ...
      canStartBuild=util.enforceChosenWorker,
  )
  

  AnyPropertyParameter

  This parameter type can only be used in properties, and allows the user to specify both the property name and value in the web form.

  This Parameter is here to reimplement old Buildbot behavior, and should be avoided. Stricter parameter names and types should be preferred.