The mechanism used by Buildbot is known as the read/write lock
Often, not all workers are equal. To address this situation, Buildbot allows to have a separate upper limit on the count for each worker. In this way, for example, you can have at most 3 concurrent builds at a fast worker, 2 at a slightly older worker, and 1 at all other workers. You can also specify the count during an access request. This specifies how many units an access consumes from the lock (in other words, as how many builds a build will count). This way, you can balance a shared resource that builders consume unevenly, for example, the amount of memory or the number of CPU cores.
The final thing you can specify when you introduce a new lock is its scope. Some constraints are global and must be enforced on all workers. Other constraints are local to each worker. A master lock is used for the global constraints. You can ensure for example that at most one build (of all builds running at all workers) accesses the database server. With a worker lock you can add a limit local to each worker. With such a lock, you can for example enforce an upper limit to the number of active builds at a worker, like above.
Time for a few examples. A master lock is defined below to protect a database, and a worker lock is created to limit the number of builds at each worker.
from buildbot.plugins import util
db_lock = util.MasterLock("database")
build_lock = util.WorkerLock("worker_builds",
maxCount=1,
maxCountForWorker={'fast': 3, 'new': 2})
db_lock is defined to be a master lock. The database string is used for uniquely identifying the lock. At the next line, a worker lock called build_lock is created with the name worker_builds. Since the requirements of the worker lock are a bit more complicated, two optional arguments are also specified. The maxCount parameter sets the default limit for builds in counting mode to 1. For the worker called 'fast' however, we want to have at most three builds, and for the worker called 'new', the upper limit is two builds running at the same time.
The next step is accessing the locks in builds. Buildbot allows a lock to be used during an entire build (from beginning to end) or only during a single build step. In the latter case, the lock is claimed for use just before the step starts and released again when the step ends. To prevent deadlocks [2], it is not possible to claim or release locks at other times.
To use locks, you add them with a locks argument to a build or a step. Each use of a lock is either in counting mode (that is, possibly shared with other builds) or in exclusive mode, and this is indicated with the syntax lock.access(mode, count), where mode is one of "counting" or "exclusive".
The optional argument count is a non-negative integer (for counting locks) or 1 (for exclusive locks). If unspecified, it defaults to 1. If 0, the access always succeeds. This argument allows to use locks for balancing a shared resource that is utilized unevenly.
A build or build step proceeds only when it has acquired all locks. If a build or step needs many locks, it may be starved [3] by other builds requiring fewer locks.
To illustrate the use of locks, here are a few examples.
from buildbot.plugins import util, steps
db_lock = util.MasterLock("database")
build_lock = util.WorkerLock("worker_builds",
maxCount=1,
maxCountForWorker={'fast': 3, 'new': 2})
f = util.BuildFactory()
f.addStep(steps.SVN(repourl="http://example.org/svn/Trunk"))
f.addStep(steps.ShellCommand(command="make all"))
f.addStep(steps.ShellCommand(command="make test",
locks=[db_lock.access('exclusive')]))
b1 = {'name': 'full1', 'workername': 'fast', 'builddir': 'f1', 'factory': f,
'locks': [build_lock.access('counting')] }
b2 = {'name': 'full2', 'workername': 'new', 'builddir': 'f2', 'factory': f,
'locks': [build_lock.access('counting')] }
b3 = {'name': 'full3', 'workername': 'old', 'builddir': 'f3', 'factory': f,
'locks': [build_lock.access('counting')] }
b4 = {'name': 'full4', 'workername': 'other', 'builddir': 'f4', 'factory': f,
'locks': [build_lock.access('counting')] }
c['builders'] = [b1, b2, b3, b4]
Here we have four workers fast, new, old, and other. Each worker performs the same checkout, make, and test build step sequence. We want to enforce that at most one test step is executed between all workers due to restrictions with the database server. This is done by adding the locks= parameter to the third step. It takes a list of locks with their access mode. Alternatively, this can take a renderable that returns a list of locks with their access mode.
In this case, only the db_lock is needed. The exclusive access mode is used to ensure there is at most one worker that executes the test step.
In addition to exclusive access to the database, we also want workers to stay responsive even under the load of a large number of builds being triggered. For this purpose, the worker lock called build_lock is defined. Since the restraint holds for entire builds, the lock is specified in the builder with 'locks': [build_lock.access('counting')].
Note that you will occasionally see lock.access(mode) written as LockAccess(lock, mode). The two are equivalent, but the former is preferred.
发布评论