Overview of e2e tests to be written

[isabel_brison]

#49507 adds e2e tests for the edit posts page. The goal is to write tests for all the admin, but we should be clear on which parts need most testing and what exactly to test. To kick things off, I'm making a list of all the pages, some of the tests we should write, and some questions that occurred to me. Feedback on what to add, change or remove is very welcome!

Dashboard:

• Quick draft section: add title, save draft, navigate to posts page and check it's there.

Updates page: not sure how we should test this one.

Categories:

• Add new category: add name, click add new category button, check if it appears in list
• Check that bulk deleting all categories leaves “uncategorized” untouched
• Add child category
• Delete category
• Search for category
• Search for non-existing category

Tags: same as categories.

Media:

• Add image to library
• Edit image alt text, title, desc
• Search for image
• Search for non-existing image

Pages: same as posts (see #49507)

Comments:

• Create post, view it, add a comment, check that it displays in comments page.

Themes:

• Search for a theme
• Activate a theme
• Delete a theme, then search for it
• Add new theme and activate it

Customize: this is pretty theme-dependent, so we should pick a theme and check a few flows on it.

Widgets and Menus: might not be worth writing tests yet as these screens are being re-written.

Theme editor:

• Add a line of CSS in the theme stylesheet and check for it in the front end?

Plugins: how should we test this page?

Users:

• Add new user
• Search for user
• change user role
• Check admin user can't change itself to subscriber

Profile: there are lots of options here, should we test changes to each of them? Some may not be relevant, e.g. profile pic just takes us to Gravatar.

Import: We could test whether the plugins install correctly, but any further than that should be unnecessary (the plugins should have their own tests)

Export/Export personal data: would it be worth trying to check the contents of the downloaded files?

Site health: there's no real interaction on this page, so maybe we can skip it.

Erase personal data: not sure how to test this one.

Settings: like profile, lots of options here. Test each individually?

[francina]

Dropping some links I mentioned this morning on Dev Chat here, so everything is under one roof :)

• Core blog post with a discussion about setup ​https://make.wordpress.org/core/2020/08/07/e2e-end-to-end-testing-in-core-working-group-proposal/
• Document with test cases created during February Yoast Contributor Day (they can be turned into instructions for automated tests) ​https://docs.google.com/document/d/1g-D3Vht0ofR7JZ3xBJiQEtdW24XTiIfzvUf5fsOJQ88/edit#heading=h.20jvhpux4c2a

Thanks :🙌

[dariak]

Hello! The overview looks very good!
Here are some of my ideas on this:

Updates page:

• expected WordPress version is shown
• Plugins, Themes and Translations sections are present

Themes (additions to already suggested use cases):

• check that the current/default Theme is active

Theme editor (additions to already suggested use cases):

• just simply add "hello world" output on a homepage (it can be easier to assert)

Plugins:

• search for a plugin, install it, activate and deactivate

Profile:

• change default data and save changes

Export/Export personal data:
I think it is enough to check that a download is triggered.

Erase personal data:

• have a test user created in setUp of test data, search for this user, and check that this user appears in Requester. I do not think the real request needs to be executed and tested on this step

Settings:
I would check here all the defaults which are filled in for all Settings options.

Trac ticket: https://core.trac.wordpress.org/ticket/52895

[justinahinon]

I am drafting here ​https://github.com/WordPress/wordpress-develop/pull/1231 the Core categories tests

Related Trac ticket: https://core.trac.wordpress.org/ticket/52895.

Related Trac ticket: https://core.trac.wordpress.org/ticket/52895.

[justinahinon]

I am going to split this ticket in multiple smaller tickets that will focus on specific parts to test.

[francina]

I believe this can then become a task, but I will defer to more experienced contributors :)

[joyously]

Customize: this is pretty theme-dependent, so we should pick a theme and check a few flows on it.

There are several core options in the Customizer, like site identity, tagline, custom CSS, menus. But the loading and saving of all the registered options is Customizer code. Also the save as draft feature is core.

Does an end-to-end test mean that the main function of the page is tested, or that all the features on the page are tested?

[hellofromTonya]

Replying to joyously:

Does an end-to-end test mean that the main function of the page is tested, or that all the features on the page are tested?

e2e testing is simulating a real world scenario to test functionality and performance from a user/product point-of-view. Click or do this workflow and then validate the expected result.

Yes, it is testing a feature and the functionality that delivers that feature. How? By running it in a complete way (end-to-end) which exercises that specific feature's software and systems to ensure all of the pieces are working together to deliver the expected behavior.

Does this make sense?