Add infrastructure for writing Playwright E2E tests #38570
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kevin940726
added
the
[Type] Automated Testing
|
Size Change: +1.57 kB (0%) Total Size: 1.21 MB
ℹ️ View Unchanged
|
kevin940726
force-pushed
the
try/playwright-test
branch
8 times, most recently
from
f39fab1
to
565c7b6
Compare
kevin940726
commented
Feb 8, 2022
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| node: ['14'] |
There was a problem hiding this comment.
We could add matrix here in the future when test suites grow and take advantage of the sharding feature.
| @@ -112,6 +113,7 @@ | |||
| "@types/eslint": "7.28.0", | |||
| "@types/estree": "0.0.50", | |||
| "@types/highlight-words-core": "1.2.1", | |||
| "@types/istanbul-lib-report": "3.0.0", | |||
There was a problem hiding this comment.
No idea why I have to add this to make tsc happy.
| }; | ||
|
|
||
| if ( isReducedMotion ) { | ||
| handleTransitionEnd(); |
There was a problem hiding this comment.
This is to fix a bug in one of the existing tests which surfaces itself during the migration. For some reason, transitionend event below never fires in CI, but I couldn't reproduce it locally. Anyway, it makes sense to skip the event when the user agent prefers reduced motion.
There was a problem hiding this comment.
This could be extracted into a separate fix; if it addresses a bug with the current tests we should get it in asap. Also it's best not to mix actual code changes with the test migration 😛
There was a problem hiding this comment.
Yep! In fact, both these two tests (customizing-widgets and new-post) are included here only for demonstration purposes, and they could be separated into new PRs along with the fix here. This should help minimize the changes introduced in this PR and make it easier to review.
packages/e2e-test-utils-playwright/src/click-block-toolbar-button.js
Outdated
Show resolved
Hide resolved
| timeout: parseInt( process.env.TIMEOUT || '', 10 ) || 100_000, // Defaults to 100 seconds. | ||
| // Don't report slow test "files", as we will be running our tests in serial. | ||
| reportSlowTests: null, | ||
| testDir: new URL( './specs', 'file:' + __filename ).pathname, |
There was a problem hiding this comment.
The code might seem weird, but it makes it easier to swap into ESM (import.meta.url) in the future if needed.
| } ); | ||
| } ); | ||
|
|
||
| class WidgetsCustomizerPage { |
There was a problem hiding this comment.
This is a Page Object Model we use in this test. We can also move this to e2e-test-utils if there are needs.
kevin940726
force-pushed
the
try/playwright-test
branch
from
8faf1d1
to
b51dbd6
Compare
mxschmitt
reviewed
Feb 11, 2022
packages/e2e-tests-playwright/specs/editor/various/new-post.spec.js
Outdated
Show resolved
Hide resolved
packages/e2e-tests-playwright/specs/editor/various/new-post.spec.js
Outdated
Show resolved
Hide resolved
packages/e2e-tests-playwright/specs/widgets/customizing-widgets.spec.js
Outdated
Show resolved
Hide resolved
packages/e2e-tests-playwright/specs/widgets/customizing-widgets.spec.js
Outdated
Show resolved
Hide resolved
|
This looks great. Looking forward to working on playwright tests. I was hoping we would have the Playwright package in WP core itself but I think because of internal dependencies of packages it would have not been feasible. |
gziolo
added
the
[Type] Technical Prototype
kevin940726
force-pushed
the
try/playwright-test
branch
from
9851ff3
to
311d9a0
Compare
8 tasks
kevin940726
requested review from
gziolo,
ntwb,
nerrad,
ajitbohra,
noisysocks,
mkaz and
ryanwelcher
as code owners
This was referenced Feb 28, 2022
This was referenced Apr 11, 2022
This was referenced Apr 23, 2022
This was referenced Sep 7, 2022
This was referenced Sep 14, 2022
This was referenced Sep 27, 2022
This was referenced Oct 20, 2022
10 tasks
This was referenced Nov 29, 2022
Closed
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Not set in stone, feedback welcome.
Follows the previous proposal and the meeting notes.
Prior art: #34089, #38401.
Notable changes
@playwright/testrather thanjest+playwright. The playwright test runner offers more features out of the box specifically for e2e testing. It's also one of the primary focuses of the playwright team (seems like they're now branding playwright as a test framework).e2e-test-utils-playwrightfor all the utils used in Playwright. The new tests written in Playwright should be intest/e2e/specs.sanity.spec.jsjust to show that it works as expected. It should be removed once we have other tests migrated.Migration steps
packages/e2e-tests/specs, rename.test.jsinto.spec.jsand put it in the same folder structure insidetest/e2e/specs.@wordpress/e2e-test-utils-playwright:describe,beforeAll,beforeEach,afterEachandafterAllwith thetest.prefix. For instance,describeis turning intotest.describe.pageandbrowser.e2e-test-utils, instead use the fixtures API to get thepageUtilsandrequestUtilsdirectly. (However,pageUtilsis not allowed inbeforeAllandafterAll, rewrite them usingrequestUtilsinstead.)e2e-test-utilsand paste it in thepageorrequestfolder ine2e-test-utils-playwrightdepending on whether it's a page util or a request util.pageandbrowserare available inPageUtils'sthis.pageandthis.browser. You can get autocomplete and type inference by adding@this {import('./').PageUtils}to the JSDoc.thisand bound to the same instance. You can remove the internal imports and just usethisto access them.index.ts, and put it inside thePageUtils/RequestUtilsclass as an instance field.pageUtilsor therequestUtilsfixture intest/e2e.Proposed best practices
Forbid
$, uselocatorinsteadIn fact, any API that returns
ElementHandleis discouraged. This includes$,$$,$eval,$$eval, etc.Locatoris a much better API and can be used with playwright's assertions. This also works great with Page Object Model since that locator is lazy and doesn't return a promise.Use accessible selectors
This PR introduces a new selector engine role-selector. It enables us to write accessible queries without having to rely on internal implementations. It's an experimental library and could be swapped out if playwright supports it natively instead. The syntax should be straightforward and looks like this:
It can also be chained with built-in selector engines to perform complex queries:
role-selectorunder the hood uses@testing-library/domto compute the query the elements and compute the accessible attributes.Selectors are strict by default
To encourage better practices for querying elements, selectors are strict by default, meaning that it will throw an error if the query returns more than one element.
Don't overload test-utils, inline simple utils
e2e-test-utilsare too bloated with too many utils. Most of them are simple enough to be inlined directly in tests. With the help of accessible selectors, simple utils are easier to write now. For utils that only take place on a certain page, use Page Object Model instead (with an exception of clearing states withrestwhich are better placed ine2e-test-utils). Otherwise, only create an util if the action is complex and repetitive enough.Favor Page Object Model over utils
As mentioned above, Page Object Model is the preferred way to create reusable utility functions on a certain page. (You can see an inline POM example in #38570 (comment))
The rationale behind using a POM is to group utils under namespaces to be easier to discover and use. In fact,
PageUtils ine2e-test-utils` package is also a POM, which avoids the need for global variables, and utils can reference each other easily.Restify actions to clear or set states.
It's slow to set states manually before or after tests, especially when they're repeated multiple times between tests. It's recommended to set them via API calls. Use
requestUtils.restandrequestUtils.batchRestinstead to call the REST API. We should still add a test for manually setting them, but that should only be tested once.Avoid global variables
pageandbrowserare not exposed as global variables, this makes it easier to work with when we have multiple pages/tabs in the same test, or if we want to run multiple tests in parallel.@playwright/testhas the concept of fixtures which allows us to injectpage,browser, and other parameters into the tests.By default,
@playwright/testwill use differentpageinstances for each test to isolate them. However, the nature of WordPress means most of our tests depend on the server state and we can't easily isolate them. We instead run the tests in serial to prevent race-condition.Make explicit assertions
We can insert as many assertions in one test as needed. It's better to make explicit assertions whenever possible. For instance, if we want to assert that a button exists before clicking on it, we can do
expect( locator ).toBeVisible()before performinglocator.click(). This makes the tests flow better and easier to read.Testing Instructions
Types of changes
New feature
Checklist:
*.native.jsfiles for terms that need renaming or removal).