Question

Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

3.5.23. test_result_set

resource type: test_result_set

Attributes:

• test_result_setid (integer) – the unique ID of this test result set

• builderid (integer) – id of the builder for this test result set

• buildid (integer) – id of the build for this test result set

• stepid (integer) – id of the step for this test result set

• description (string) – Free-form description of the source of the test data

• category (string) –

  The category of the test result set. This describes what data the test results contain.

  Any value is allowed. The following standard categories are defined:

  • pass_fail: The test result set contains results that can indicate success or failure of specific test. The values of test results contain success or failure values.

  • pass_only: The test result set contains results that can only indicate success of specific test. This is used in cases when failed tests are not reported.

  • fail_only: The test result set contains results that can only indicate failure of specific test. This is used in tests when passed tests are not reported.

  • code_issue: The test result set contains issues within the code reported by various tooling. This is effectively a subset of fail_only.

  • performance: The test result set contains performance results. The values of test results contain some kind of performance metric such as time per operation or the number of operations completed in a time period.

  • binary_size: The test result set contains evaluation of binary size. The values of test results contain a binary size metric.

  • memory_use: The test result set contains evaluation of dynamic memory use. The values of test results contain a memory use metric.

• value_unit (string) –

  Describes the unit of the values stored within the test results.

  Any value is allowed. The following standard units are defined:

  • ps: Picoseconds

  • ns: Nanoseconds

  • us: Microseconds

  • ms: Milliseconds

  • s: Seconds

  • boolean: A boolean value (0 or 1)

  • B: Bytes

  • KB: Kilobytes (1000-based)

  • KiB: Kibibytes (1024-based)

  • MB: Megabytes (1000-based)

  • MiB: Mebibytes (1024-based)

  • GB: Gigabytes (1000-based)

  • GiB: Gibibytes (1024-based)

  • TB: Gigabytes (1000-based)

  • TiB: Gibibytes (1024-based)

  • message: Arbitrary string message

  Note that the value of the test result is always stored as string.

• tests_passed? (integer) – The number of passed tests in cases when the pass or fail criteria depends only on how that single test runs. For example, performance tests that track regressions across multiple tests do not have the number of passed tests defined.

• tests_failed? (integer) – The number of failed tests in cases when the pass or fail criteria depends only on how that single test runs. For example, performance tests that track regressions across multiple tests do not have the number of failed tests defined.

• complete (boolean) – true if all test results associated with test result set have been generated. Once set to true this property will never be set back to false

example

{
    "test_result_setid": 412,
    "builderid": 14,
    "buildid": 31,
    "stepid": 3,
    "description": "Performance test via BenchmarkDotNet",
    "category": "performance",
    "value_unit": "ms",
    "complete": true
}

This resource represents a test result set. A test result set consists of a number of related test results. These test results need to be related in that they represent the same type of data and are produced by a single step. In reasonably tested codebases the number of test results in a test result set will approach several or even tens of thousands.

There may be a long delay between the creation of the test result set and full creation of the corresponding test results. This is tracked by the complete property. If it’s true, then the full set of test results have been committed to the database.

The test_result_unparsed_set object tracks test result sets that have not been parsed yet.

3.5.23.1. Update Methods

All update methods are available as attributes of master.data.updates.

class buildbot.data.test_result_sets.TestResultSet

path: /builders/{builderid_or_buildername}/test_result_sets

Path Keys:

builderid_or_buildername (number|identifier) – the ID or name of the builder

This selects all test result sets that have been created for a particular builder

GET

returns

collection of test_result_set

path: /builds/{buildid}/steps/{step_number_or_name}/test_result_sets

Path Keys:

• buildid (number) – the id of the build

• step_number_or_name (identifier|number) – the name or number of the step

This selects all test result sets that have been created for a particular step

GET

returns

collection of test_result_set

path: /builds/{buildid}/test_result_sets

Path Keys:

buildid (number) – the id of the build

This selects all test result sets that have been created for a particular build

GET

returns

collection of test_result_set

path: /steps/{stepid}/test_result_sets

Path Keys:

stepid (number) – the id of the step

This selects all test result sets that have been created for a particular step

GET

returns

collection of test_result_set

path: /test_result_sets/{test_result_setid}

Path Keys:

test_result_setid (number) – the id of the test result set

Selects a test result set by id

GET

returns

collection of test_result_set