WPT coverage

The library is tested against the web-platform-tests (WPT), the conformance test suite shared by the browsers, at the revision 5602b0f210e3. This page gives the pass rate of each section of the test suite. The page of a section lists its tests, with the reason why each test that does not pass fails.

The tests are split in three groups:

• Stable tests: the testharness.js tests that are not marked as tentative: they test the behaviors defined by a specification (some of these specifications are still drafts or incubations). They measure the conformance of the library.
• Tentative tests: the testharness.js tests of the experimental features and of the proposed changes that no specification requires yet (the WPT convention: .tentative. in the file name or a tentative directory). They may change or be removed, and browsers often do not pass them: they are reported separately and are not part of the stable coverage.
• Reftests: the reftests (tentative ones included), which compare the rendering of a page with the rendering of its references, and the print reftests, which compare the pages of the printed documents: they measure the layout, the pagination and the painting.

The columns:

• Tests: the testharness.js test files of the section that run in a window or in a dedicated or shared worker (or in a service worker for the tests of service-workers) and the reftests, according to the WPT manifest of the revision, and the other files that are run (a file with several variants counts once: a .any.js test passes when it passes in all its globals, a reftest when all its variants pass). Visual tests, crash tests, manual tests, WebDriver tests, the tests that only run in a ShadowRealm, and the tests outside of service-workers that only run in service workers are not counted: the library has no ShadowRealm, and the service workers are only tested by the tests of service-workers.
• Passing: the testharness.js tests whose harness status is OK and whose subtests all pass with the JavaScript engine (Meziantou.Framework.JavaScriptEngine), and the reftests whose rendering matches their references.
• Disabled: the tests that are not run: they are skipped, or their directory is not part of the tested directories. The page of the section gives the reason.
• Pass rate: the passing tests among the tests that are run (the tests that are not disabled).

The pages of the sections give, for each test that does not pass, the Kind and the Description of the reasons why it does not pass (a test can have several reasons), and the Failure reasons section of each page (this one included) counts the tests that do not pass per kind and per feature. The kinds:

• BUG: the implementation is broken.
• TEST BUG: the test is wrong: it does not match the specification.
• JS ENGINE: a limitation of the JavaScript engine (Meziantou.Framework.JavaScriptEngine) or of its integration with the library.
• NOT IMPLEMENTED: the feature is not implemented yet.
• REQUIRES(features): the feature cannot be implemented, or the test cannot pass, until other features are implemented (listed in parentheses).
• TEST RUNNER: the WPT runner cannot run the test: a .py handler of the WPT server is not emulated, or the directory is not fetched.
• OUT OF SCOPE: the library will not support it.
• FLAKY: the test passes or fails randomly.
• PERFORMANCE: the test is too slow to be run by every build.

The features of REQUIRES:

• animations: CSS animations and transitions, Web Animations.
• hardware: devices and OS integration: sensors, cameras, screens, Bluetooth, USB.
• layout: a layout engine: boxes, scrolling, hit testing.
• media: audio and video playback, media decoders and encoders.
• service-workers: service workers.
• user-input: user input: pointer, keyboard and wheel actions, user activation.
• web-storage: Web Storage (localStorage, sessionStorage).
• workers: dedicated and shared workers.
• worklets: worklets.

Summary

Failure reasons

The tests that do not pass (failing or disabled), by kind of reason. A test with several reasons counts once per kind.

• BUG: 174
• TEST BUG: 1,033
• NOT IMPLEMENTED: 4,252
• REQUIRES: 762
  • hardware: 574
  • layout: 20
  • media: 20
  • service-workers: 1
  • user-input: 154
  • workers: 3
• TEST RUNNER: 309
• OUT OF SCOPE: 205
• FLAKY: 17
• PERFORMANCE: 7

Stable tests

Tentative tests

Reftests