Shadow DOM traversal support

[43081j]

I've been gradually trying to push support of shadow DOM v1 into selenium (see #4230, #5762).

However, we are still missing one major piece: traversal.

There needs to be some ability to traverse shadow DOM when calling FindElement and what not.

So maybe we can discuss some possible implementations here?

What we can't/shouldn't do:

• Alter how existing selectors work (CSS, XPath, etc. they should all behave as they do now)
• Implement deprecated things like /deep/
• Implement our own selector syntax/mechanism

The way I see this working is that we should treat the DOM as we would in the browser, meaning there will not be a way to select a deep element in one call:

element
  .FindElement(By.CssSelector("foo"))
  .ShadowRoot?
  .FindElement(By.CssSelector("bar"));

Which, in the browser, would be structured the same (no shortcuts):

element
  .querySelector('foo')
  .shadowRoot
  .querySelector('bar');

So maybe we just need to implement ShadowRoot on WebElement? Which can be null (just like in the browser).

[p0deje]

cc @jleyba @juangj @barancev @shs96c

[43081j]

also, while i do disagree with us implementing some kind of "shortcut", something like this could work:

element
  .FindElement(
    By.ShadowSelector(
      By.CssSelector("foo"),
      By.CssSelector("bar")
    )
  );

which is the equivalent of element.querySelector('foo').shadowRoot.querySelector('bar').

i can imagine there are plenty of people who want a "lazy" way to do whats in my original post, without the need for null checking and what not. Basically a single FindElement call to achieve the same as the expanded form.

either way i think we still need to implement ShadowRoot though at least.

[p0deje]

Even though I don't fully follow the request, I see nothing in regards to shadow root in specification. To my knowledge, support for custom commands like ShadowRoot is required to be implemented in underlying drivers and so has to be a part of specification.

[43081j]

The request is to provide shadow DOM support. To provide a way of using FindElement to reach nodes contained within shadow DOM trees.

Currently, you do not have any support for shadow DOM at all in regards to traversing the tree.

So if we can't add ShadowRoot, what do you suggest?

[AutomatedTester]

Originally we had been discussing this mostly in the same approach as frames so you would do.

shadow_root = driver.find_element(...)
driver.switch_to_shadow_root(shadow_root)
driver.find_element(....).click()

This is how I have recently been implementing in Firefox as Firefox's shadow DOM becomes live.

Then if you wanted to traverse back up the tree you would do

shadow_root = driver.find_element(...)
driver.switch_to_shadow_root(shadow_root)
driver.find_element(....).click()
# Traverse back to the main tree
driver.switch_to_shadow_root(None)

[43081j]

A solution like that makes sense to me too, its a very similar way of tackling it to my original idea (ShadowRoot property).

Though I do think there should be a way of accessing a root relative to an element...

Just like you can do Element.FindElement or Driver.FindElement.

You don't always have the original driver instance which created the element you are being passed, so how do you reach a shadow root from only a WebElement?

Of course there are ways around that (refactor so you do have a ref to the driver), but it seems to me it would be quite nice to have access through the element, being that is what "owns" it.

Right now many people have to resort to executing a script which returns arguments[0].shadowRoot essentially.

[jimevans]

@43081j I’ll just point out that in Java st least (and .NET), you do almost always have a reference to the parent driver. You can usually cast to WrapsDriver and use getWrappedDriver() (or something similar; I may have the syntax slightly wrong there).

[43081j]

@jimevans really didn't realise there was a reference to the driver there 🙈

in that case, its fair enough, having a switch_to_shadow_root makes a lot of sense.

though do remember, find_element right now can't get a direct ref to the shadow root. so are you saying switch_to_shadow_root takes the element as a parameter? (not the shadow root its self)

[AutomatedTester]

@43081j shadow root is a property of an element so passing in the element and then looking for the property makes sense to pass in the webelement.

The think that gets tricky is when we hit the issue of the closed shadow DOM vs what we do with an open shadow DOM.

[43081j]

yup sounds good to me.

as for closed shadow roots, i think it should just be accepted that you can't access it (throw an exception).

reason being, shadowRoot in the DOM its self will be null. so that way our method is following the same behaviour. as far as i know, there's no way to access it without a reference from the initial attachShadow call, which we will/can never have really...

so if we call switch_to_shadow_root and shadowRoot is null, throw an exception? because that covers both closed roots and non-existent roots.

[AutomatedTester]

Yes, we need to throw an exception when shadowRoot is null, I am currently throwing NoSuchElement though I am not married to that solution. I am happy to see about adding a new exception type if need people want that

[43081j]

a new exception would make more sense i think, seeing as its not an element really. its a document fragment i guess.

so either an exception specific to shadow roots, or one a little more generic tied to documents (NoSuchDocument, NoSuchFragment, NoSuchShadowRoot, etc but better worded).

i dont think we need to care about closed roots as we shouldn't be able to access them anyway. which makes the logic as simple as "is it null or not".

[p0deje]

If I understand correctly, we need to make it a part of spec (like frame switching), don't we? Or can we implement it purely in client bindings?

[AutomatedTester]

Part of the spec. We originally punted this because Shadow DOM was chrome only and the L0 version was crap and no other browser wanted to implement it. Since that is ancient history and shadow dom is in most browsers (at least in their nightly builds) we can move forward with what we have discussed here. I am happy to write the spec prose

…

On Tue, May 15, 2018 at 3:37 PM, Alex Rodionov ***@***.***> wrote: If I understand correctly, we need to make it a part of spec (like frame switching <https://w3c.github.io/webdriver/#switch-to-frame>), don't we? Or can we implement it purely in client bindings? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5869 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH2Bogm-h_sKkV3NGqDqt-ue-5rwbuMks5tyuhDgaJpZM4T3Dip> .

[43081j]

@AutomatedTester if you're happy to, could you get the ball rolling on this if you're happy to write the spec?

would be great to get some movement on this

[AutomatedTester]

I will do it, I am just waiting for the spec to reach spec on Thursday, David Burns Email: david.burns@theautomatedtester.co.uk URL: http://www.theautomatedtester.co.uk/

…

On Tue, May 29, 2018 at 7:35 PM, James Garbutt ***@***.***> wrote: @AutomatedTester <https://github.com/AutomatedTester> if you're happy to, could you get the ball rolling on this if you're happy to write the spec? would be great to get some movement on this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5869 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH2BgduQpayjXWrVfXOPDO0xXbLclu3ks5t3ZTkgaJpZM4T3Dip> .

[43081j]

is there any update on this?

i do still question if the switching idea makes sense though, as it feels a little unnatural to be accessing an element's state/property in an indirect manner. its a shame we can't simply have element.shadowRoot in the spec...

cc @rictic and @justinfagnani

[AutomatedTester]

I have been busy with other things but I have most of the spec written up locally.

I am not sure what you mean by accessing it indirectly?

To get an element state or property in a shadow DOM your proposal seems to suggest you want to do

element.shadowRoot.element.getProperty()

Now if you want do multiple actions in the shadow DOM you now have to repeat constantly that you want to do element.shadowRoot. before actually doing anything. This may be alright but the repetition makes me incredibly sad.

The next problem is if you have 2 elements with the same ID, one in the main document and one in the shadow DOM and you forget to do the repetition of moving state you could start getting into false positives.

The idea of switching helps us, and the client code, maintain a state that the code author thinks they are in and tries to remove a footgun. There is also the fun in knowing if it really is a shadowRoot or not.

[justinfagnani]

element.shadowRoot.element.getProperty()

Now if you want do multiple actions in the shadow DOM you now have to repeat constantly that you want to do element.shadowRoot. before actually doing anything. This may be alright but the repetition makes me incredibly sad.

This is just the DOM APIs, or really any API with nested objects, and there are plenty of other places where you might have a value that you need to dereference multiple time.

Presumably you can deal with this like any other nested object and define a variable:

const shadowRoot = element.shadowRoot;
shadowRoot.findElement(By.name('query')).sendKeys('hello');
shadowRoot.findElement(By.name('search')).click();

The repetition of shadowRoot doesn't actually stand out, and its cost, if any really, isn't worth creating an API that differs so much from the actual DOM structure, IMO.

The idea of switching helps us, and the client code, maintain a state that the code author thinks they are in and tries to remove a footgun.

Switching seems really fraught to me. When is state restored? What if a test forgets to switch back, or if there's an exception? Will other new DOM APIs be put behind global switching state? HTMLTemplateElement#content is another example of a branch in the DOM tree that's not part of childNodes, does that need switching?

Also, a ShadowRoot is simply a type of Node that's not included in childNodes, but it's otherwise in the DOM tree like any other Node. Switching seems like a high conceptual and ergonomic barrier compared to getting other common nodes like Elements.

[AutomatedTester]

element.shadowRoot.element.getProperty()

Now if you want do multiple actions in the shadow DOM you now have to repeat constantly that you want to do element.shadowRoot. before actually doing anything. This may be alright but the repetition makes me incredibly sad.

This is just the DOM APIs, or really any API with nested objects, and there are plenty of other places where you might have a value that you need to dereference multiple time.

Presumably you can deal with this like any other nested object and define a variable:

const shadowRoot = element.shadowRoot;
shadowRoot.findElement(By.name('query')).sendKeys('hello');
shadowRoot.findElement(By.name('search')).click();

So we're talking about the WebDriver API and not just DOM APIs. Not all DOM APIs map directly to webdriver because that wouldn't make sense. The idea is to conceptually treat the shadow DOM in the same way we treat frames.

This would make it like

driver.switch_to_shadow_root(element)

# switching a frame would be
driver.switch_to_frame(element)

Yes we could make it something like

shadow = element.shadow_root
shadow.find_element(....).send_keys("I like cheese")

But that now causes shadow to look like a WebElement, which yes it is because it's document.documentElement equivalent but clashes with the semantics of moving to different "state". Also if the shadowRoot is removed from that node there is the potential that instead of getting a new NoSuchShadowRoot error you would get a StaleElementReference.

Switching seems really fraught to me. When is state restored? What if a test forgets to switch back, or if there's an exception? Will other new DOM APIs be put behind global switching state? HTMLTemplateElement#content is another example of a branch in the DOM tree that's not part of childNodes, does that need switching?

Also, a ShadowRoot is simply a type of Node that's not included in childNodes, but it's otherwise in the DOM tree like any other Node. Switching seems like a high conceptual and ergonomic barrier compared to getting other common nodes like Elements.

WedDriver authors are already used to the idea that they need to be conscious of their state in their tests if they had iframes. I don't see how this is different. The idea is if you you are in a shadow DOM to get back to the top document you would simply do

driver.switch_to_shadow_root(None)

which is equivalent frame switching would be

driver.switch_to_frame(None)

[justinfagnani]

The idea is to conceptually treat the shadow DOM in the same way we treat frames.

I think a huge difference between ShadowRoots and frames is the sheer pervasiveness and number of them. Apps are likely to have 100s to 1000s of ShadowRoots, basically every Custom Element will.

And they're not nearly as special as iframes - they're in the same window and document, share the same globals, many style properties inherit through them, some events bubble up past them, light-DOM children are projected into them, and they're always attached to a host.

Many tests need to check on some interaction between elements in the ShadowRoot and outside of it - ie, set a property on the host and make sure it's reflected in the shadow; click on a button in the shadow and many sure that an event is fired on the host, or an attribute is added; add a child and make sure it's assigned to a slot, style a CSS ::part and check that it's applied, etc. The idea that a ShadowRoot is a different state just doesn't really hold. It's merely a different branch in the DOM tree (the tree of trees).

Iframes on the other hand have none of these APIs/interactions, and interactions between the iframe and it's host document are very limited, so these types of tests are probably non-existent and the pain that a switching-based API would cause haven't been felt.

ShadowRoots also very commonly nest, whereas with frames testing through arbitrary testing nesting levels is probably pretty rare. So we'll want the ability to traverse deeply into ShadowRoots, possibly multiple branches of the tree of trees in a single test to test cross-components interactions.

Consider a search form with three custom components - a text field, a button, and an output area - that all have shadow roots, all composed inside a shadow root.

One way the test looks like this:

const root = element.shadowRoot;
root.findElement(By.css('x-text-field')).shadowRoot.findElement(By.css('input')).sendKeys('hello');
root.findElement(By.css('x-button')).shadowRoot.findElement(By.css('button')).click();
assert.equals(root.findElement(By.css('x-output')).shadowRoot.getText(), 'hello');

Another way is like this presumably:

driver.switchToShadowRoot(element);
driver.switchToShadowRoot(driver.findElement(By.css('x-text-field')));
driver.findElement(By.css('input')).sendKeys('hello');
driver.switchToShadowRoot(element);
driver.switchToShadowRoot(driver.findElement(By.css('x-button')));
driver.findElement(By.css('button')).click();
driver.switchToShadowRoot(element);
driver.switchToShadowRoot(driver.findElement(By.css('x-output')));
// I'm not sure how to do this - how would we get non-Element Nodes from a Driver instance?
assert.equals(driver.getText(), 'hello');
// And make sure you switch back at the end of the test!
driver.switchToShadowRoot(element);

I personally find that much harder to follow because of the internal state of the driver, and there are certain things that are just not possible because you can't get a reference to a ShadowRoot, ie, you can't pass a root to a helper function, you're IDE won't help identify what root you're referencing on a line because it's not a reference...

Also, now that we're using WebDriver to interact with ShadowRoots, what do all the other APIs on WebDriver do? Why do I care about get(url), getCurrentUrl(), getTitle(), etc.? These can only return the window's state, but it's confusing to me at least to intermix global state like that and the state of an individual element. In contrast, these APIs do apply to Iframes.

Also if the shadowRoot is removed from that node

It's not possible to remove a ShadowRoot, even if you delete the shadowRoot reference. The only way a ShadowRoot is destroyed is if its host is - so the danger is exactly the same as if an Element was removed from the document.

[dominikg]

I'd like to second @justinfagnani.

Nested ShadowRoots are going to be very common and how would you e.g. select a list of all .whatever elements in a scenario like this where custom-list, custom-child and foo all have their own shadowroots?

<custom-list>
  <custom-child>
    <foo>
      <span class="whatever">x</span>
    </foo>
  </custom-child>
  <custom-child>
    <foo>
      <span class="whatever">y</span>
    </foo>
  </custom-child>
  <custom-child>
    <foo>
      <span class="whatever">z</span>
    </foo>
  </custom-child>
</custom-list>

[gsnedders]

Related: w3c/csswg-drafts#640

[AutomatedTester]

I have written a spec proposal in w3c/webdriver#1320

[justinfagnani]

@AutomatedTester you didn't address any of my points in my last comment. Shadow DOM simply isn't like iframes, as I've pointed out, and some of your concerns, like removing a ShadowRoot, just aren't possible. The switching API will add significant and unnecessary overhead to testing Web Components.

[AutomatedTester]

@justinfagnani how the client API looks doesn't have to mirror the webdriver spec exactly so we can sort that as we go.

The idea is fundamentally is where we are working from and we need to "switch" to that point. Since there is no way to from document down to the nth level shadow root without switching to shadowRoot. This is particularly important when it comes to the interactability checks that webdriver does.

In the case of a DOMElement being a child of a component that is n levels down when we call document.elementsFromPoint(...) we will get the top most shadowRoot. If we then go down and call myShadowRoot1.elementsFromPoint(...) we then get the next shadowRoot.

Since we have this situation where the remote end needs to know what context it should be in we need to tell it to switch to that and carry on doing the next item. As @gsnedders pointed out we dont have a mechanism to penetrate shadowRoots as this was dropped by the CSSWG.

[Karnauj]

Hello to all,
I'm currently working with Shadow Rooting and when trying to extend the Page Factory functionality realized something. Wouldn't it make more sense to change the element's Search Context? And treat it's component holder (as the last shadow root) as the origin of the context?

Regarding the handling of the Shadow Elements themselves, I see two major sources of concerns:

• The shadow element might change it's position in the Shadow Tree in any moment, making the location of the Shadow Element itself very brittle
• Different Shadow Components treats their elements independently, so it's likely that at some point several Shadow Elements are going to possess elements with the same locators

In that regard, I see that for the first issue a search would be recommended to avoid having to specify the whole tree of elements. But in that case, the second issue would raise the issue of collisions, forcing the framework to allow certain flexibility for the user to adapt to its certain case.

Would it make sense to add the following functionalities?

• Include one special ShadowElement just to allow the transversing of the Shadow Tree, implementing capabilities to transverse the dom/shadow tree (not implementing WebElement)
• Add an extension to the WebElement so a shadowElement can be referenced for its location. This specification could enable concatenation of locators (for example, parentShadowComponent.itsParentShadow) to specify whichever length of the tree as required

Thanks

[corevo]

@MelbinFrancis you are barking up the wrong tree, this is not a bindings issue, rather a W3C issue as @AutomatedTester said, the proposal will have to be accepted first.

[msn-pixel]

Has there been any movement on this? Or does anyone know of a viable Selenium alternative that supports ShadowDOM well?

I was able to get the Shadow DOM to work with chrome, but no success with SafariDriver since Chromedriver now supports shadowDOM but not SafariDriver

[Tallyb]

Need shadow dom support - I have one word for you: Playwright.
All problems gone...

[dominikg]

playwright does not offer on-device testing in mobile browsers and is not an option for downstream issues like appium/appium#9733.

I'm really surprised that this hasn't been fixed in over 2 years and that there's not more activity here. Is this really such a niche/edge case that not many run into? Or do they simply not use selenium for this kind of testing at all?

[Varsha176]

@msn-pixel Can you please share the approach you followed. It would be really helpful.

[wrharper]

I actually have a 6 layer nested shadowroot with constant elements changing to deal with. In my opinion by far the iwebelements need to be able to have a .shadowroot for cleaned / easier to handle code. The current way is a flat out nightmare. Executescript needs to also be smart enough to continue off of a webelement. In fact I think executescript is the key for you to link all this together!
Keep executescript open ended and combine with all other elements. This should be more than possible!

[justnpT]

I believe the original idea was very good, and I would be happy to see this topic pushed forward.
Shadow-dom is indeed a game-changer for some selenium-based test-automation frameworks.
Currently selenium operations within shadow-dom are very limited and the only way to get full selenium support is to never use shadow-dom on the front-end. But if front-end really wants to use it, then there is a problem. This is at least my experience

[Tallyb]

@justnpT Maybe it is not appropriate to mention it here, but for in our project, it was reason #1 to move to Playwright (https://playwright.dev). It has everything you could dream of for shadow dom support.

[justnpT]

@Tallyb this is an interesting solution for anyone who is ok with building js-based test framework and drop all of the features that selenium environments and community brings in. It is very good that it is mentioned here as alternative.

[ngenereux-ps]

It appears that the webdriver has been updated to include shadow DOMs by @AutomatedTester . What is the next step in adding support to Selenium?

[AutomatedTester]

@ngenereux-ps Well, we need an implementation from a browser vendor. I will try to carve out time over the next couple of weeks to add it to a browser so we can see how it works.

[wrharper]

Finally! This is heavily needed in my c# programming.

Does this mean we can finally stop parsing .ExecuteScript?

[justnpT]

it looks like this shadow-dom tech took selenium by surprise and changed the game a little bit 😆 lucky those who don't have to deal with it

[Tallyb]

@justnpT just as a note, Playwright now has bindings for:
JavaScript and TypeScript
Python
Java
C#
That seems to cover a big part of the community needs.

[Eli-Black-Work]

Glad to find some discussion here about Selenium supporting ShadowDOM 🙂

Echoing @justinfagnani and @dominikg:

For our project, I'm looking into supporting scoped CSS, so that CSS from one module doesn't spill over into other modules. Support for <style scoped> has been dropped from the HTML spec., and W3C's current position appears to be that it's better to use ShadowDOM for scoped CSS (w3c/csswg-drafts#3547)

If that's the case, then I think it's likely that there might be a lot of nested ShadowDOM elements. In our case, we'd probably have almost one ShadowDOM per HTML element:

<div className="card">
  #shadow-root
    <div className="filter">
      #shadow-root
        <div className="filter-block">
          #shadow-root
            ...
          #shadow-root-end
         </div>
      #shadow-root-end
    </div>
  #shadow-root-end
</div>

This is just our current concept, so we may find out that this is a bad idea 🙂 If we do implement something like this, though, I think it would make testing with Selenium almost impossible, given the current ShadowDOM API.

Not sure what the fix is for this (and as I mentioned, we might find out that this doesn't work for our needs, anyway), but thought I'd throw this out there 🙂

[jimevans]

We should probably close this issue, as shadow DOM traversal has been added to both the W3C spec and to (most? All of?) the language bindings. In Java, the WebElement interface gets a getShadowRoot() method, which returns a SearchContext (the interface containing findElement() and findElements()). If the element does not have a shadow root, a NoSuchShadowRoot exception is thrown.

The exception throwing behavior is consistent with the Selenium philosophy that you should know the structure of the page you’re automating. If the page does not match that understanding, that is an exceptional condition, warranting throwing an exception.

[justnpT]

Maybe there is some link to the pull request / topic where new changes werre introduced ?
This thread got closed, but what was added is just the syntax for opening shadow root or the support for xpath inside and other things described in the original request description ?

[diemol]

It got closed because the implementation on the Selenium side is ready, what is missing is the implementation on the browser drivers side.

[daluu]

So that means folks should open up tickets with the respective browser drivers that are missing the implementation? If so, be good to link them here for reference.

[titusfortner]

The implementation work is already underway. Chrome v96 support is likely:
https://wpt.fyi/results/webdriver/tests

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.