web-platform-tests 编辑

web-platform-tests is a regression test suite covering standard Web platform features. It is shared both among browser vendors and across browser platforms (such as Mozilla's Gecko and Servo), and is one of the primary means of verifying that browser implementations correctly implement the standards and are interoperable. Because of this, adding new tests to web-platform-tests (as opposed to Mozilla-specific Reftests, Mochitests, etc. suites) is encouraged whenever writing tests that might sensibly be run in multiple browsers.

The web-platform-tests suite consists of three test types: (1) JavaScript tests (to test DOM features, for example) written using the testharness.js library, (2) reference tests (to test rendered output with what's expected to ensure that the rendering is done properly) written using the W3C reftest format, and (3) WebDriver specification tests written using pytest.

Mozilla has integrated web-platform-tests into mozilla-central, and has a two-way sync mechanism that allows developers to add/modify tests directly in the testing/web-platform directory in the Gecko source. (Changes are automatically synchronized between the Gecko source and the upstream GitHub repository.)

Running tests

Like other test suites, web-platform-tests are invoked using the mach command:

./mach wpt

Individual tests can either be run by path in the source tree or by name (i.e. path under the server root). For example, the following two commands perform the same test:

./mach wpt testing/web-platform/tests/dom/historical.html

./mach wpt /dom/historical.html

One can also pass the path to one or more directories to run all tests under that directory.

Many of the same considerations that apply to mochitests/reftests apply to web-platform-tests; for instance, there may be tests that require focus.

Running under a debugger

The tests can be run under a debugger using the same flags as for other test suites. For example: --debugger=gdb or --debugger=lldb.

The --pause-on-unexpected flag can also be useful to pause on the first failing test allowing you to attach a debugger and reload the test to debug it.

Logging

web-platform-tests use structured logging, and all the usual structured logging options are available.

From a running browser

You can also run tests directly from an already-running browser instance. This can be convenient when rapidly iterating on a test, or to use the browser's built-in JS debugger. To do this, first launch the web-platform-test harness as a service, like this:

./mach wpt-serve

Then in your browser, run your test by navigating to either:

http://web-platform.test:8000/dom/historical.html

Or for https:

https://web-platform.test:8443/dom/nodes/Document-createEvent.https.html

Note that you may get invalid certificate errors when doing this, and browsers may stop you from proceeding. In Firefox there is still an option to proceed buried under "Advanced". In Chrome, you may need to specify chrome://flags/#allow-insecure-localhost and run it under:

https://localhost:8443/dom/nodes/Document-createEvent.https.html

Writing tests

General guides can be found on Test the Web Forward. Mozilla-specific concerns follow.

Where to put new tests

New tests with pass conditions that match the relevant specifications and that should be upstreamed and shared with other browsers should be added under testing/web-platform/tests.

Tests that are (currently) Mozilla-only should be added to testing/web-platform/mozilla/tests. (Note that tests in this directory will have /_mozilla/ prepended to their URL.)

Test directories can contain a mix of both reftest and testharness.js tests.

Note that wdspec tests must live under webdriver/.

Creating new tests

The fastest way to add a new test is to use the mach addtest command. It will create a new test file with test boilerplate, and opens it in your local $VISUAL or $EDITOR (if defined),

testharness.js example:

./mach addtest testing/web-platform/tests/spec/new_test.html

Creating a new reftest is similar, but pass the -r or --reftest arguments, and optionally specify the reference file using the --reference argument (also opened in your editor if it doesn't exist). For example:

./mach addtest testing/web-platform/tests/spec/new_test.html --reference testing/web-platform/tests/spec/new_test_ref.html

Metadata

Unlike for Mozilla's reftest and mochitest test suites, there are no human edited reftest.list or mochitest.in files alongside web-platform-tests tests. Test metadata common to all browser implementations (such as the reference files for a reftest tests) is stored in the test files themselves. Implementation specific metadata (such as pass/fail expectations) is stored outside the web-platform-tests tests directory.

In the past, the manifest file containing the list of tests was stored in-tree and required manual steps to update when tests were added or changed. This file is no longer stored in tree, and instead a recent version is downloaded on demand and updated to match the local checkout.

Because web-platform-test files are synced with upstream and do not use a human-written manifest, it's necessary to keep implementation-specific metadata outside the tests themselves. In Mozilla's case, we store metadata that we don't share with other implementations under testing/web-platform/meta/ and testing/web-platform/mozilla/meta. This is done in a metadata with the same relative path as the test relative to the test root, but an .ini extension e.g. testing/web-platform/tests/dom/historical.html has a metadata file under testing/web-platform/meta/dom/historical.html.ini. In addition, metadata can be set for a whole subtree of tests using a metadata file called __dir__.ini under the metadata root of that subtree (e.g. testing/web-platform/meta/dom/__dir__.ini).

Despite the .ini extension, these files are not true ini files, since they allow nesting and conditional logic on values.

Expectation data

Because web-platform-tests are intended to check the standard behavior rather than the behavior of specific implementations, it is common for tests to fail in specific implementations. Therefore, the expected result of each test that doesn't pass is kept in the corresponding metadata file. A simple example of such a file might be:

[filename.html]
    type: testharness

    [Subtest name for failing test]
        expected: FAIL

    [Subtest name for erroring test]
        expected: ERROR

It is possible for tests to be updated with multiple statuses, if preferred. When expected is a list, the first status is the primary expected status and the trailing statuses listed are known intermittent statuses.

[filename.html]
   type: testharness
    expected: [PASS, TIMEOUT]

Expectations can be made platform-specific using a simple set of python-like conditions. For example, for a test that times out on Linux and fails on other platforms:

[filename.html]
   type: reftest
    expected:
        if os == "linux": TIMEOUT
        FAIL

The available set of condition variables is that provided by mozinfo plus a boolean variable e10s that is true when e10s is enabled.

Expectation data can be updated automatically on the basis of a test run (e.g. from try), using the raw structured log files. On treeherder, these are files listed as artifact uploaded: wpt_raw.log. Locally they can be generated using a mach command like:

./mach wpt testing/web-platform/tests/dom/historical.html --log-raw=historical.log

Then to update the expectation data:

./mach wpt-update historical.log

By default, this will make a commit (in a git tree) or an mq patch (in an hg tree). To override this behavior, use --no-patch. If the expected result of a test is platform-specific, you must provide multiple logs—one for each platform—to the update command, or the correct metadata will not be generated. If you want to wipe away the existing metadata for the test rather than try to update the conditions in place (such as when a test goes from having platform-specific behavior to generic behavior) use --ignore-existing.

Disabling tests

You can disable a test by adding a disabled key to the metadata file. The value can be anything except the special values @False and @Reset, but by convention is the bug number detailing the reason the test was disabled. For example:

[filename.html]
   type: testharness
    disabled:
        if os == "win": https://bugzilla.mozilla.org/show_bug.cgi?id=1234567

Note: If you disable a test as a whole, it will be skipped entirely. If you disable a subtest, the subtest will still run, but the results will be not cause the test to identify as a failure.

When updating test metadata with ./mach wpt-update, it is possible to disable tests that have inconsistent results across many runs using --disable-intermittent.  This can precede a message providing a reason why that test is disable. If no message is provided, unstable is the default text.

Recording intermittent statuses

To update test metadata with known intermittent statuses, the --update-intermittent option can be used. The expected key stores known intermittent statuses in addition to the primary expected status. The default behaviour of this option is to retain any existing intermittent statuses in the expected list unless --remove-intermittent is also specified. When using remove in addition to the update option, any obsolete intermittent statuses that did not occur in the specified logfiles are removed from the list.

Setting prefs

Per-test prefs can be set using a metadata item called prefs which takes a list of preferences to set and the values to assign to them. This lets tests run against features which are preferenced off by default, for instance.

[filename.html]
    prefs: [dom.serviceWorkers.enabled:true,
            dom.serviceWorkers.interception.enabled:true,
            dom.serviceWorkers.exemptFromPerDomainMax:true,
            dom.caches.enabled:true]

Per-directory settings

A __dir__.inifile creates metadata that applies to the entire subtree beneath it. For example, we can disable all DOM tests using a file in testing/web-platform/meta/dom/__dir__.ini file:

disabled: http://some-bug

Having done this, we might need to re-enable a specific test under dom/. This can be done using the special value @False in the metadata file for that test:

[filename.html]
    disabled: @False

With lists of preferences, the preferences applied are typically those for the test and those from any __dir__.ini files up to the test root. If we need to apply a different set of prefs to a specific test, the special value @Reset prevents all inheritance of prefs from further up the tree.

Running tests in other browsers

Tests can be run in other browsers by using the --product argument to mach wpt e.g.

./mach wpt --product chrome testing/web-platform/dom/historical.html

Currently supported values for --product are servo, chrome, and edge. For Chrome and Edge the product-specific WebDriver binary is required; for Chrome the mach command will prompt to download this, but for Edge you will have to download the correct WebDriver for your specific Edge version. Since expectation metadata is not shared across products, the default metadata directory for non-Firefox browsers is testing/web-platform/product/<product>/. This directory is part of the VCS ignore list, so you are free to add any expectation data you find helpful.

Firefox for Android (GeckoView)

You can run the tests against a Gecko-based browser (GeckoView) on an Android emulator. As shown below, to do so you must start an emulator, build Firefox for Android and then run mach wpt with the org.mozilla.geckoview.test package. The package will be installed interactively by mach and tests will run against TestRunnerActivity.

./mach android-emulator --version x86-7.0
./mach build && ./mach package
./mach wpt --package=org.mozilla.geckoview.test

Servo

In addition to support for running via mach wpt in a gecko checkout, Servo has its own import of web-platform-tests, with tests living under tests/wpt/web-platform-tests/. It also has mach commands similar to those provided in the Gecko build system, with slightly different names. In particular, to run the in-tree copy one can do this:

./mach test-wpt

in a Servo checkout.

Lint

web-platform-tests has associated lints, that check for two things:

1. Test file correctness/style issues
2. Manifest correctness

The lints can be run using mozlint e.g.

mach lint -l wpt -l wpt_manifest

Test Correctness

This lint tests for common mistakes made when writing web-platform-tests and style issues.

Examples of problems detected using this lint include:

1. Incorrect links to testharness.js and testharnessreport.js
2. <meta name=timeout content=long> appearing in an incorrect place in the file
3. Trailing whitespace in test files
4. Use of windows-style line endings

In rare cases it may be necessary to allow some code that this lint would ordinarily reject (e.g. a test that relies on a literal CR LF pair in the source). In this case the file testing/web-platform/tests/lint.whitelist must be updated with the error code and the path that should be excluded from linting.

Manifest Correctness

This lint checks that the wpt manifest is up to date. If this lint fails the fix is always to run:

mach wpt-manifest-update

and commit the resulting changes to MANIFEST.json files.

Repeating

If you are experiencing intermittent failures, some of the following might help:

--repeat N

Repeat N number of times.

--repeat-until-unexpected

Continue running a test until it fails unexpectedly (could be forever).