Francesca Marano 7:49 am on August 7, 2020
Tags: core-test ( 9 )

E2E (End-To-End) Testing in Core Proposal

What

E2E (End To End) tests are a form of automated software testing which runs in the context of a user using an application. The intention is to test the complete “end to end” workflow that a user would take in various scenarios.

In web software development this means running and controlling a web browser pointing at a web page, either fully automatically by a machine, or by a user following the test steps.

Frameworks for running E2E tests allow a web browser to be controlled fully automatically, for instance by navigating to URLs, filling out forms, and clicking on links and buttons. The tests operate by checking assertions about the expected content or behaviour of the page and triggering a test failure when they are not met.

Why

Testing individual aspects of an application’s code — for example with unit tests or integration tests — reduces the chance of low-level bugs in the code that powers an application. Unit testing helps maintain the quality of the code that powers the application.

In contrast, E2E testing tests the higher-level functionality of an application that the user sees, typically including the interface and the inputs and controls that the user interacts with.

Unit testing and E2E testing can, should, and do co-exist.

How

E2E tests are usually human-readable and provide logical steps for a user to perform. For example:

Given I am logged in as an Author
When I view the list of posts in the dashboard
And I see posts published by other users
Then I should not see an “Edit” link next to posts which are published by other users

The scenario above can potentially form a completely automated E2E test, depending on the framework in use and the built-in steps that have been defined for it. The tests are written in such a way that a human can follow the same steps and come to the same result.

This means anyone, not just a developer, can write an outline of an E2E test.

Implementing E2E tests in WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.

WordPress Core right now has only one E2E test in place, to check that the wp-adminadmin (and super admin) page loads. GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/, on the other hand, has quite a few.

There are some tickets and pull requests open about this:
https://core.trac.wordpress.org/ticket/43395
https://core.trac.wordpress.org/ticket/49507

There is also a test environment but it is not currently being used as part of the automated test suite: https://make.wordpress.org/core/2019/08/05/wordpress-local-environment/

It would be beneficial to create a project to work on this in a structured way in order to maintain the stability and reliability of WordPress core as new features are added and changes are made.

Moving Forward

Months ago, this topic was mentioned multiple times and @pandjarov from SiteGround has offered to set up the test environment plus some engineering time to work on the tests.

We could either set up a new one or check that the existing one is still the right way forward.

To really kick things off, the Core Team has to come up with requirements for the test environment, its tests, and the preferred workflow, so the tests are really useful.

What is needed

A list of requirements – I think Core Committers, especially with a long history of contributing (the so-called historians) are best suited to start the list with the most pressing concerns
Evaluating the existing test environment (existing vs creating new)
Ideas about what functionality in WordPress should be a priority in the automated tests, for example, the upgrade process, or managing users, or installing plugins
Developers who are familiar with Jest, which is the test framework in place in Gutenberg and WordPress core, so the test environment can be improved, and to write new tests
A fork of the wordpress-develop repo on GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ where the project can be managed and PRs opened
Volunteers to work on this. It’s a big task because it starts from 0 – this is very important because what the previous post about this has shown, in my opinion, is that an environment is not enough 😉
Coordination with the Test team
For making it easier to communicate I propose to add a dedicated weekly chat in the #core-test channel – From September 6th I am available to do that.

Deadline to comment

To make sure that the project lifts off I think it needs enough interest and volunteers to actively work on it.

Please leave a comment before September 6th (so you have a month to think about it!) with the requirements and/or (possibly AND) your availability to be part of this.

Riad Benguella 8:30 am on August 7, 2020

Thanks for starting this. I agree there needs to be a coordinated effort for the first tests but what I’d like to see is a culture shift where we could start suggesting to include e2e tests in patches for bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. fixes that hard to unit tests…

A culture shift needs buy-in from coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. committers.
- John Blackbourn 12:14 pm on August 7, 2020
  
  I fully support this.
- Frank Klein 10:57 am on August 10, 2020
  
  I agree as well. Happy to help with this.
Janz Nordman 3:23 pm on August 10, 2020

Codeception can be used here. Works quite well. Can be extended also for WP specific needs.
- Jonathan Bossenger 6:34 am on August 16, 2020
  
  Apologies for hijacking this post slightly, but I would like to add my recommendation for the use of Codeception, or for the consideration of a Codeception like tool for adding functional tests to coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., to be included in our testing efforts.
  
  Please excuse me is this comes across as mansplaining, but automated tests are typically divided into three categories, unit tests, functional tests and acceptance tests.
  
  Acceptance tests (aka E2E tests), as @francina has pointed out, test “the higher-level functionality of an application that the user sees”. Example: The “logged in as an Author” user story described above.
  
  Unit tests already exist in core, and test individual units of code, example does the function that verifies access permissions return true if I am an author.
  
  Functional tests, which fit somewhere in the middle, test an entire section of functionality, but without the requirement of the user facing portion. Example, if I send a GET request to the ListPosts action, logged in as an Author, does it verify my access permissions, and return the relevant Posts Lists response data, in order to render the relevant view.
  
  Functional tests typically take place on the same technical level as unit tests, but test larger pieces of functionality, like E2E tests. They also don’t require the testing tool to spawn instances of headless browsers to test the functionality in question, and are therefore quicker to run.
  
  Functional tests also help test the underlying functionality of the acceptance tests separately from the user interface, meaning tracking down complicated bugs is simplified. So, if an acceptance test fails, but the functional test passes, we can determine that the problem lies in the user interface layer, and not the functional layer, and focus our debugging efforts there.
  
  Using something like Codeception also means that the tests can be written in PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher, a language that many WordPress developers are already used to, and this means it lowers the barrier for many folks to assist with testing efforts.
  
  There is one (large) consideration, and that’s the fact that to include functional testing means to possibly have to refactor sections of the code base, to make it possible for different sections of functionality to be tested separately.
  
  I’m not 100% sure if this has been discussed and decided upon at any point in time? It might also not make sense to include functional tests in an acceptance test proposal, and a separate proposal might be required, or even a separate post, asking these questions (which I’m more than happy to do, if there is agreement).
  
  But it starts with a discussion, and I thought the platform where adding more automated tests to core is already being discussed, might be a good place to gauge interest in adding functional tests to core as well.
  - Sylvain Schellenberger 2:51 pm on September 23, 2020
    
    Seems I’m a little late to the party… I like that you wanted to define what we are talking about, and I agree with your definition except that I’ll be very wary with the functional tests terminology.
    
    I fear its quite for people to confuse the level of the tests (~ the amount of code that is tested), with the type of the tests (~ what should these tests proof?).
    
    If you conform to the [ISTQB definition](https://www.istqb.org/downloads/send/2-foundation-level-documents/281-istqb-ctfl-syllabus-2018-v3-1.html), functional tests are a type of tests that focuses on verifying that the system performs the task it should perform. As opposed to non-functional tests which evaluates technical characteristics of a software, or piece of software, like performance or security.
    
    The level of type you described are referred in the mentioned document as integration tests.
    
    The types of the tests doesn’t imply of specific level, as you can test security of a component (non-functional unit testunit test Code written to test a small piece of code or functionality within a larger application. Everything from themes to WordPress core have a series of unit tests. Also see regression.), of the data transmission between several components (non-functional integration test) and of the user inputted data (non-functional system test). As well as you can test that the blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. parser retrieves the correct block in a content (functional unit test), that the serialized block are inserted in the database (functional integration test) and that the user can write a post containing several blocks (functional system test).
    
    In the end, it doesn’t need to be official naming convention, as long as we understand each other.
    
    But on the topic of test levels, I cannot leave without mentioning Mike Cohn’s Pyramid of Tests (example: MountainGoatSoftware: The forgotten layer of the test automation pyramid) which strongly advocates towards restricting ourselves from writing too many high level tests, because of their high cost.
- midemike 10:34 pm on August 20, 2020
  
  Following!
Daniel Richards 3:15 am on August 12, 2020

I’d be very happy to help with this. I have a fair bit of experience with the existing test framework and started writing some tests a while ago for the Edit Posts/All Posts screen (https://github.com/WordPress/wordpress-develop/pull/200), which I’ll try to pick up again.

Something not mentioned on the list that would be good to have is documentation on how to write tests, and general philosophies for e2e testing, but perhaps that can come after a good direction has been set by the initial discussions.
- Mike Schroder 9:40 am on August 19, 2020
  
  Thanks so much for putting that together!
Brett Krueger 8:13 pm on August 12, 2020

I would love to work on this if volunteers are needed. Readable and reproducible tests are extremely important and I feel like this would be a great way to contribute while learning-by-doing. I’m still fresh enough that I don’t have many ideas to contribute for what should be tested, aside from all of the basic functions like making a new post, editing a post, (which it looks like @talldanwp already did), adding/modifying users and capabilities, pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party installs, potentially APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. responses etc.
Mike Schroder 9:38 am on August 19, 2020

Same. Happy to help with this!

Requirements is tricky, because I think adding tests for *anything* would be helpful — but it also gives a lot of room for contribution!
Helen Hou-Sandi 10:23 pm on August 20, 2020

Something I’ve been playing with recently that’s a layer up from this is dev containers in VSCode, which also map to the GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ Codespaces feature that’s in betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process.. I can describe that more fully if needed, but basically it’s config to spin up a Docker container with a given image, either locally using VSCode or remotely in Codespaces (which is on Azure, in case you look this up and notice the name overlap). The remote component is of particular interest to me because it connects an in-browser VSCode editor to the container, meaning that you can edit code and view the results without needing to have a specific setup locally. I think this is a big deal in terms of continuing to broaden the contributor base, overcoming some economic barriers and making it easier to get started. I also think it would be good for whatever’s done to encourage pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party and theme developers to use the same environment, write tests, and run them against trunktrunk A directory in Subversion containing the latest development code in preparation for the next major release cycle. If you are running "trunk", then you are on the latest revision..

I’m definitely not eager to push for something that ties closely to another ecosystem like that, but I think there’s at least some amount of value in keeping that kind of broader use in mind when it comes to the environment. I don’t know how this fits into existing environments/tools like wp-env right now, but I’m happy to talk with people to get a better understanding of it all and see where I can find some other people with time to work on it.
Monika Rao 11:50 am on September 3, 2020

Thanks for sharing. It’s really interesting and tricky but I would love to work on this.
This type of Automation testing would be very helpful in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress..

Just as additional information in here, I found this set of tests from Pantheon that’s pretty cool — I like how easy to read Behat is, too.

https://github.com/pantheon-systems/pantheon-wordpress-upstream-tests/tree/master/features

E.g. from https://github.com/pantheon-systems/pantheon-wordpress-upstream-tests/blob/master/features/posts.feature

Feature: Manage WordPress posts

  Background:
    Given I log in as an admin

  Scenario: Create and publish a blog post
    When I go to "/wp-admin/post-new.php"
    And I fill in "post_title" with "Awesome Post"
    And I fill in "post_name" with "awesome-post"
    And I press "publish"
    Then print current URL
    And I should see "Post published."

    When I go to "/awesome-post/"
    Then print current URL
    And I should see "Awesome Post"

The list of what’s being tested in here might help with the initial requirements.
It also makes me wonder what the tradeoffs (if any) are with Behat, as it looks like it might make it easier for folks to contribute tests than what is used here (Jest, I believe):
https://github.com/WordPress/wordpress-develop/blob/master/tests/e2e/specs/hello.test.js

import { visitAdminPage } from '@wordpress/e2e-test-utils';

describe( 'Hello World', () => {
	it( 'Should load properly', async () => {
		await visitAdminPage( '/' );
		const nodes = await page.$x(
			'//h2[contains(text(), "Welcome to WordPress!")]'
		);
		expect( nodes.length ).not.toEqual( 0 );
	} );
} );

Francesca Marano 10:51 am on September 24, 2020

Thank you for all your replies.
This is what I am hearing so far:

1. Some commenters proposed a platform change. The issue with it is that we need someone to own it, and with Jest we are covered by the GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ team. So, unless there is a very compelling reason to switch to any other platform (and someone wants to own it), it seems to me that sticking to Jest would be the best option.
2. We need a test plan. @monikarao, our 5.6 Test Focus Lead, is working on some first steps and we will start test scrubs soon. This part is a work in progress.
3. We need people to write the tests. This step will come once number 1 is confirmed and number 2 is less vague.

Is my recap accurate?
Marking this as Top Comment because clearly to decide on number 1 we need a different discussion 🙂

Comments are closed.

Welcome!

Communication

E2E (End-To-End) Testing in Core Proposal

What

Why

How

Implementing E2E tests in WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.

Moving Forward

What is needed

Deadline to comment

Welcome!

Communication

What

Why

How

Implementing E2E tests in WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.

Moving Forward

What is needed

Deadline to comment

Share this: