-162

Update Mar 21: TL;DR: Today, we graduated the experiment for Stack Overflow question pages with zero answers, specifically the variant where three related questions are shown by default, with a link to view more.

During the experiment, we observed a statistically significant 900%+ increase in clickthrough rates (CTR) in both experiment groups. Users in the experiment were engaging with related questions at an exponential rate compared to those in the control group where related questions were shown in the sidebar.

Update Feb 24: As of Feb 23, testing has concluded

Update Feb 14: Testing is now active

Background

As part of the Content Discovery initiative, we’re announcing the next experiment. In our previous experiment to reorder related questions, we mentioned that most users are not seeing related questions above the fold. We also believe that banner blindness is a common web behavior and pattern, wherein users have a high tendency to ignore all forms of advertising that typically appear in the sidebar.

We believe displaying related questions within the answers list will allow users to get to the most relevant content sooner. If users typically ignore the sidebar, this new option could allow them to naturally discover related questions. In this post, we explain the goals of the experiment, success criteria, and how we will conduct the test.

Experiment goals and success criteria

Given that this is our first attempt to display additional content within the main content area, we’re interested in learning how users will engage with it and whether this will provide any incremental value when trying to find relevant content to get closer to their just-in-time needs.

Our null hypothesis is that engagement with related questions remains the same in both experiment groups. We will determine whether the variant is a winner if there is a statistical significant lift to clicks on related questions, thus allowing us to reject the null hypothesis.

How we’re conducting the test

The experiment will be conducted on Stack Overflow question pages for anonymous users only. It will target 20% of the traffic that will be split evenly between the control and two variant groups.

We learned from recent user research that users tend to check for other answers and comments right away after reading a question and try solutions offered in Q&A. If users aren’t able to find an answer, we think they are more likely to bounce as this prevents them from being able to test potential solutions. We are initially focusing our efforts on questions without answers in order to validate whether the inline placement results in better engagement before we iterate on question pages with at least one answer.

Here are screenshots of how related questions will appear in each variant group. In the first variant, users will see three related questions below the question post with an option to view more by clicking on the “See more related questions” link. In the second variant, users will see up to five related questions below the question page. Users who are opted into either variant group will only be seeing inline related questions on question pages with no answers. The sidebar related questions on those pages will also be hidden. Question pages with at least one answer will not be impacted by this experiment. Users in the control group will continue to see related questions in the sidebar.

Three related questions displayed below the question post with a link to view more

Figure 1: Three related questions displayed below the question post with a link to view more

5 related questions displayed below the question post

Figure 2: Up to 5 related questions displayed below the question post

The experiment will run for at least a week in order to account for the newness effect. Once the experiment concludes, we will analyze the results and share them with Meta.

50
  • 45
    Has there been any research on how often related questions are... even remotely useful in terms of solving a problem one might have for a question that has no answers? or is this strictly for reducing bounce %.
    – Kevin B
    Commented Feb 10, 2023 at 16:29
  • 24
    Why not make the sidebar more useful so people don't ignore it?
    – samcarter_is_at_topanswers.xyz
    Commented Feb 10, 2023 at 16:36
  • 13
    I don't think it's necessarily a bad change, from a UI perspective... but I doubt it's going to be a useful change for people looking for a solution unless the question is literally a duplicate that should have been caught before submit.
    – Kevin B
    Commented Feb 10, 2023 at 16:36
  • 42
    Is this an improved list or just moving what's in the sidebar? Too often those questions are either the most basic questions in the same tag that aren't related to what you're doing or they're the same or a similar question in an unrelated language.
    – Laurel
    Commented Feb 10, 2023 at 16:42
  • 36
    While this is a location that gives more visibility to related questions, once again you are wasting time improving minor things while leaving important ones untouched. You should optimize asking questions and showing more relevant questions when people ask, but to do that you need to move specifying tags under the question title, so that you stop showing irrelevant questions in other languages and platforms because you search only based on title.
    – Dalija Prasnikar Mod
    Commented Feb 10, 2023 at 18:41
  • 20
    Re "displaying related questions within the answers": That is like Quora. Without some very prominent visual markers, it it very confusing to the reader.
    – Peter Mortensen
    Commented Feb 11, 2023 at 2:57
  • 24
    I am not entirely sure how the related questions search runs, but it definitely needs improvement. It doesn't matter how many related questions are shown if they are not properly attuned to what should be related to the actual post. Search needs to be improved, maybe look to some existing work relating to search. Modifying the accuracy of the related question will be orders of magnitude more effective than showing more mildly related questions.
    – Travis J
    Commented Feb 11, 2023 at 7:04
  • 16
    "We believe displaying related questions within the answers list will allow users to get to the most relevant content sooner." Only if related questions gave relevant content, which it doesn't, because the search facility is so bad. (So bad that I wonder whether any of you researching have even tried to evaluate what benefit the related questions has or even tried to use it.)
    – philipxy
    Commented Feb 13, 2023 at 19:24
  • 37
    So i take it we've decided to run with this and launch it? using the existing algorithm that presents mostly useless results?
    – Kevin B
    Commented Mar 21, 2023 at 18:25
  • 14
    I think they should put more effort into getting relevant content into the list before caring about the UI aspects. As long as the content of the list is in fact unrelated, people will ignore it, no matter where it is placed. For the time being, please hide the list.
    – jps
    Commented Mar 21, 2023 at 18:45
  • 13
    @jps It's worse than that. I used to find some value (though usually indirect value) in the list of related questions (sometimes even as indirect as "oh, this other question also needs to be closed). Now, with it in such a disruptive position, I will be forced to hide the entire section with a userscript, meaning I will have zero chance of deriving value from it even if the content it shows improves.
    – TylerH
    Commented Mar 21, 2023 at 18:52
  • 33
    wtf is the point in clickthrough rates if they're not clicking through to anything useful?
    – Nick is tired
    Commented Mar 21, 2023 at 19:11
  • 21
    How do I get rid of it?
    – Paolo
    Commented Mar 21, 2023 at 19:11
  • 32
    Please, please, please stop just editing questions with "updates" that change the entire demeanor of the post. This is not a forum. The change to turn this on is drastically different than the original discussion, and there is no way to answer the now live version of this with issues or other problems observed in the wild. Releasing this as a feature should have its own post, which describes the results of the A/B test, gives some sort of nod to community feedback (or any feedback), explains expected outcomes, and allows for a discussion on the widespread integration.
    – Travis J
    Commented Mar 22, 2023 at 6:10
  • 20
    I think it's hysterical that the "fix" for low click through rate on the "related questions" was not to make the "related questions" actually related, but to instead shove them in our faces :D If this is the "fix", I'd suggest you further improve it by changing the section title to Completed Unrelated Questions.
    – devlin carnate
    Commented Mar 22, 2023 at 20:30

17 Answers 17

Reset to default
173

Gah!!!

This change is counterproductive, unless the selection of "related" posts is substantially improved.

Before making the "related posts" section be in your face, the selection of "related" questions needs to be substantially improved to at least take the question's tags into account. There are multiple places where the entire site experience would be far better if the list of automatically shown "related" questions took the current question's tags into account. Fixing that overall is what should have time allocated to it, not making very poor selections of "related" questions more "in your face".

Changing the location and visibility of these is counterproductive when the selection of "related posts" is often completely useless. The reason people, at least experienced users, don't click on them when they are in the sidebar is because we usually just ignore that section, given that the "related posts" are commonly not related at all to the current question.

Once the selection of "related" questions is fixed, then it would be reasonable to look at changing where they are displayed.

Changing these to being more "in your face" will increase users' frustration in a lot of circumstances (e.g. "the site can't even show me questions that are actually related"). Not having good selections will also result in user's just learning to ignore the new in-your-face "related questions" section, which is counterproductive to, eventually, having an actually good selection of "related questions". If the selection isn't improved first, then when/if they are, you have to work against all the training you've done for your users to ignore that section.

You had a 900%+ increase in click-through rates, but is that relevant?

It appears you graduated this "feature" primarily because of the dramatic increase in click-through rates. That the click-through rates are dramatically increased is absolutely expected. If you make something more obvious, then people are going to click on it more. The real question is if those clicks actually helped your users. Given the near-universally poor relevance of the links that are shown, due to poor "related question" selection, I'd argue that a large percentage of the people that did click through ended up wasting their time looking at questions which were not, in fact, related and weren't relevant to solving their issue.

The primary factor which this should be evaluated is: Did the users' experience improve (i.e. did they find a solution to the problem they are having)?

The problem with evaluating it on that basis is that you'd have to measure some secondary factors as surrogates for the links being helpful, given that a notable number of people will just give up looking on SO, or at least with SO search/"related" links, once they realize that what they are being presented with has a low probability of being useful.

Ryan M has some good suggestions for metrics in comments (1, 2, with some reformatting):

Metric suggestions:

  1. % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question).
  2. % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help).
  3. % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources).
  4. A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed.

#5 provided by Donald Duck in a comment:

  1. % of users who, after landing on a page with a related questions module and clicking on one of the related questions, upvoted the related question or one of its answers (good: higher, showing that more users found the related question helpful).

#6 provided by Ian Campbell in a comment:

  1. Percentage of users who land on a related question page and then copy a portion of that page (higher better, because they copied that code to use it). I think you keep up with copies, right?

Please improve the selection of "related questions" before making this change.

Note: I don't like the new position for this section, so have a userscript which changes it back, thanks to work by Samuel Liew and released under a MIT license (code in commit 1 and 2), which is included in his ReduceClutter userscript (see his answer below for more details on that userscript).

18
  • 23
    "... is bad for all your users and for Stack Overflow." tis great for ad impressions tho,
    – Kevin B
    Commented Mar 21, 2023 at 19:34
  • 21
    Emphatic Gah!!! indeed.
    – Andras Deak -- Слава Україні
    Commented Mar 21, 2023 at 21:38
  • 3
    Metric suggestions: 1) % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question). 2) % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help). 3) % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources).
    – Ryan M Mod
    Commented Mar 22, 2023 at 2:25
  • 1
    A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed.
    – Ryan M Mod
    Commented Mar 22, 2023 at 2:29
  • 1
    Userscript already exists :)
    – Samuel Liew
    Commented Mar 22, 2023 at 3:04
  • 3
    @SamuelLiew Your userscript seems helpful for many people. Thanks for working on it. The overall script does more than just this change. For me personally, at least the last time I checked, some of the changes it makes I didn't want and/or conflicted with userscripts I already use. My use is a bit of a special case, given the number of userscripts I already run and that I make many custom modifications that have developed over time. :; I appreciate that you've already done the work. I'll probably end up copying the relevant code, obviously with attribution. Thank you for making it available.
    – Makyen Mod
    Commented Mar 22, 2023 at 3:16
  • 4
    Suggestion for another metric: 5. % of users who, after landing on a page with a related questions module and clicking on one of the related questions, upvoted the related question or one of its answers (good: higher, showing that more users found the related question helpful).
    – Donald Duck
    Commented Mar 22, 2023 at 15:21
  • 3
    We really appreciate this feedback. We are taking all of this into consideration, and have already been in the process of preparing another Meta post that addresses most of these concerns for tomorrow. Also, thanks for the metrics you suggested! This gives us some ideas on which we can build some funnels within our event tracking platform to analyze. We are adding a lot more metrics to capture engagement from when users click on related questions so we will have a lot more insights to share.
    – Bella_Blue StaffMod
    Commented Mar 22, 2023 at 18:12
  • 4
    The focus on click rates on themselves and not user satisfaction makes me think about the trust thermocline, which was already brought under my attention by the advertisements for technology courses.
    – The_spider
    Commented Mar 22, 2023 at 20:30
  • 4
    Suggestion for another metric: Percentage of users who land on a related question page and then copy a portion of that page (higher better, because they copied that code to use it). I think you keep up with copies, right?
    – Ian Campbell
    Commented Mar 22, 2023 at 21:43
  • 1
    I can't agree more. I've seen quite a few questions recently where the list of "Related Questions" aren't related at all, and most likely just add to the frustration of the OP. They certainly add to the frustration of a person wanting to answer the questions.
    – Dawood ibn Kareem
    Commented Mar 23, 2023 at 0:04
  • @IanCampbell 's suggestion is great, because there is apparently code in place already to detect copy-paste
    – Charlieface
    Commented Mar 23, 2023 at 0:47
  • I think all of the proposed metrics fail in fairly significant ways: 1. Many users might not even have an account or desire to ask. E.g., they might bounce around links to find information. It's how I search and I've observed others IRL do the same. The information needed might even be found elsewhere, e.g., GitHub or a blog post, etc. At which point it wasn't SO who helped but there would be no question the user asked. 2. Plenty of users delete their questions early. I have no idea how to link this to Related. I've seen questions deleted seconds after being posted regularly.1/5
    – VLAZ
    Commented Mar 23, 2023 at 8:12
  • 1
    helpful. If it wasn't helpful and the question still wasn't answered (due to complexity, or requiring an expert in a narrow field, or missing information, or plain because it fell through the cracks and wasn't noticed). 5. would fail to account for irrelevant but eye-catching questions. If a user lands on a question about Foo but sees a Related one about something which is not Foo but still interesting, they might visit and attempt to upvoted answers. The upvoting might be a "Thanks, I liked that" or "I am an expert and I approve of this". Where in all cases the user would vote based 4/5
    – VLAZ
    Commented Mar 23, 2023 at 8:13
  • 12
    @Bella_Blue "We are taking all of this into consideration, and have already been in the process of preparing another Meta post that addresses most of these concerns for tomorrow" - unless that post announces the immediate rollback of this bad change, it will probably not address any of the concerns in a reasonable way.
    – l4mpi
    Commented Mar 23, 2023 at 11:40
99

First, this is a bad change. It requires users to scroll further down to see related content compared to when it was on the sidebar, making that section even more useless, especially given how narrow the content column of the site design is. Not to mention the main, unsolved problem with Related Questions: the criteria for what shows up in that list is woefully bad/not useful. Now that bad/not useful content is even harder to ignore.

Please revert it.

Second, why is the test re-activated? I am seeing this today:

enter image description here

6
  • just a note, it puts it further down the page for desktop, and higher on mobile
    – Kevin B
    Commented Mar 21, 2023 at 18:28
  • 26
    Agree that it's bad. It basically highlights the fact that the related questions query mostly returns junk. (What is the difference between String and string in C#? is NOT the answer to every c# question.)
    – dbc
    Commented Mar 21, 2023 at 18:36
  • 4
    Related on MSE: Why are related question suddenly showing up above the answer input instead in the right sidebar on posts with no answers?
    – starball
    Commented Mar 21, 2023 at 18:36
  • 7
    For the adblocker folks looking for an immediate solution, the IDs to block are #inline_related_var_a_less, #inline_related_see_more, and #inline_related_see_less. This will kill the entire related list and not replace it, but that's a trade-off y'all are gonna have to decide on individually
    – Zoe Mod
    Commented Mar 21, 2023 at 19:00
  • 13
    Question was edited after you posted this: "Today, we graduated the experiment for Stack Overflow question pages with zero answers, specifically the variant where three related questions are shown by default, with a link to view more". Good thing I had already downvoted this question. This is a mind-bogglingly intrusive and useless feature to the point of being harmful. The "related questions" are useless so shoving them in people's faces is just ugh.
    – Andras Deak -- Слава Україні
    Commented Mar 21, 2023 at 21:34
  • 2
    Also an official answer I almost missed (hence this comment). We have clicks now, phew 😌
    – Andras Deak -- Слава Україні
    Commented Mar 21, 2023 at 21:51
61

This is not useful for people answering questions

I kind of see the value of this feature for people coming from search engines, looking for a solution to their problem (setting aside the quality of the suggestions).

But it isn't useful at all for people who mostly answer the questions, such as myself.

And since we tend to monitor the site for new questions, we're going to see this very often, as most of them aren't yet answered.

Perhaps a setting to move the list back to its original position (to the sidebar) would be useful. Or some heuristic to not show this to people who mostly answer questions.

2
  • 5
    (If the links really were to possible duplicates & people were reasonably given motivation to flag new questions as duplicates ...) It is useful to answerers in that they should not be answering questions that already have answers in the list but should be flagging as duplicate.
    – philipxy
    Commented Mar 22, 2023 at 17:32
  • 2
    The reason that this isn't useful to answerers (who, as @philipxy points out, should be trying to identify duplicates rather than answering them) is generally that the suggestions are bad, rather than anything specific to answerers vs askers.
    – Ryan M Mod
    Commented Mar 23, 2023 at 17:14
50

Positioning this list so prominently is completely nonsensical if the contents of the list are completely unrelated.

Fix that first, before changing the layout.

In this example, the user is asking for the difference between 2 APIs, and the list is showing a load of completely unrelated JS questions.

Looking at that list, I suspect that:

  • Way too much weight is given to question score.
  • Any question that starts with "What's the difference between..." is automatically related.

enter image description here

3
  • 33
    Clippy: "You seem to be interested in JavaScript, why don't you go read our most popular JavaScript questions?"
    – CodeCaster
    Commented Mar 22, 2023 at 11:35
  • 13
    "...the contents of the list are completely unrelated. Fix that first, before changing the layout." is spot on. It's stunning that the OP has explicitly stated in a comment "There are good questions to ask about how to serve better related questions but that's out of scope here". Why "out of scope"???!!! Finding truly relevant questions is by far the hardest part of this project. If that can't be done then everything else is just rearranging deckchairs on the Titanic. How can they expect to obtain meaningful metrics when what they will be measuring is not meaningful?
    – skomisa
    Commented Mar 23, 2023 at 6:07
  • 2
    The "out of scope" comment is particularly strange considering that the goal is ostensibly to increase engagement with these links. Surely there would be more engagement if the links were more relevant?
    – kaya3
    Commented Mar 24, 2023 at 9:07
45

During the experiment, we observed a statistically significant 900%+ increase in clickthrough rates (CTR) in both experiment groups. Users in the experiment were engaging with related questions at an exponential rate compared to those in the control group where related questions were shown in the sidebar.

Why was this metric chosen? Were any other metrics monitored, such as whether this resulted in questions being more likely to receive answers, or the amount of time spent scrolling past this box while writing answers?

If click-through-rate is the sole metric by which changes like this are to be judged, then Stack Overflow will become useless fairly quickly. Any link will get a higher CTR if you put it in the middle of the page and make it bigger, but of course moving everything to the middle and making everything bigger will not improve the site for anybody.

This is certainly not the first time that a change in Stack Overflow's UX has been poorly received because the motivation for that change was wrong in the first place.

3
  • 11
    "Any link will get a higher CTR if you put it in the middle of the page and make it bigger" also: the title of the section is misleading. It says "related questions" so a user landing after a Google search would be deceived into thinking these questions are indeed related. Where, in a lot of cases, the only relation is the same language tag. A user deceived by this title may be disappointed after visiting the link. But that can't really be measured by CTR. Moreover, the CTR itself may be inflated: even clicking the background will navigate away when this could be an easy mistake.
    – VLAZ
    Commented Mar 22, 2023 at 6:35
  • 1
    Makyen's answer and the comments quoted in it discuss some metrics that would likely have been better choices.
    – Ryan M Mod
    Commented Mar 23, 2023 at 17:16
  • 3
    Exactly. This is like arguing that lamp posts are more useful in the middle of the road than off to the side, because more motorists collide with them there.
    – Dawood ibn Kareem
    Commented Mar 25, 2023 at 3:50
40

During the experiment, we observed a statistically significant 900%+ increase in clickthrough rates (CTR) in both experiment groups. Users in the experiment were engaging with related questions at an exponential rate compared to those in the control group where related questions were shown in the sidebar.

Let me rephrase that:

"If we put links right into people's faces, they start clicking them to see what the fuss is about".

Can we puh-lease not follow financial experiments to the detriment of the public benefit akin to Facebook, LinkedIn, BuzzFeed, TikTok, Reddit and many more, just to keep people scrolling and clicking? We've got enough addiction problems in youth already.

"Engagement" is not the holy grail, however hard your investors try to tell you that it is.

Or, in the legendary words of Sterling Archer: "Do you want people to start blocking your elements? Because that's how you get people to start blocking your elements." These are the ids for blocking the related block in uBlock/EasyList format:

! 2023-03-22 https://stackoverflow.com
stackoverflow.com###inline_related_var_a_less
stackoverflow.com###inline_related_var_a_more
stackoverflow.com###inline_related_see_more
stackoverflow.com###inline_related_see_less
4
  • Makyen's answer and the comments quoted in it discuss some metrics that would likely have been better choices.
    – Ryan M Mod
    Commented Mar 23, 2023 at 17:17
  • 1
    You mean the metrics suggested by you, @Ryan? :P I've upvoted that answer.
    – CodeCaster
    Commented Mar 23, 2023 at 19:04
  • 1
    I only suggested 4 of the 6 :-) I can't take all the credit.
    – Ryan M Mod
    Commented Mar 23, 2023 at 21:17
  • I hadn't used this feature until now, guess I'll never know if they resolve the issue ;)
    – pilchard
    Commented Mar 24, 2023 at 21:44
39

It feels wrong(? not sure what word to use) to put things that are not intended to be answers in the space dedicated for answers. I've seen my share of "related" posts that are hardly even related, and I've heard my share of people saying that the related posts listing is generally less useful than the linked posts listing.

Putting myself in the shoes of an asker who's familiar with the site, I think I would find it confusing. And as someone who primarily writes answers, it feels like an eyesore. A toggle setting would be nice, similar to how we have one for hot network questions.

If anything, shouldn't we be doubling down on getting clicks for duplicate-suggestions in the Ask-A-Question UI if the problem is that there are posts with answers to questions with no answers?

3
  • 9
    "I've heard my share of people saying that the related posts listing is generally less useful than the linked posts listing." - I mean, to be fair, that's to be expected. Linked posts are posts that have been specifically linked by people who presumably put some thought into it.
    – Nick is tired
    Commented Mar 21, 2023 at 18:43
  • 9
    Also worth noting that the effort that went into the algorithm for the related questions is extremely low. It takes minimal effort to get an even vaguely more relevant link
    – Zoe Mod
    Commented Mar 21, 2023 at 19:04
  • 3
    @ZoestandswithUkraine just ask ChatGPT to recommend related questions.
    – Andras Deak -- Слава Україні
    Commented Mar 21, 2023 at 21:39
27

I have published a userstyle on Stack Apps which removes the Related Questions from the middle of the page.

For posterity:

See the code on GitHub (Direct installation)

In case you never used userstyles:

You need a userstyle manager which will add the CSS to the pages it is configured for. I use Stylus which is available on all major browsers.

If you prefer to use an element blocker (most adblockers have a built-in one) then the selectors needed are #inline_related_var_a_less, #inline_related_see_more, and #inline_related_see_less.

2
  • 4
    Works great! Thanks a lot for the quick workaround!
    – samcarter_is_at_topanswers.xyz
    Commented Mar 21, 2023 at 19:27
  • 3
    I made a userscript to move it back to the sidebar instead, see my newer answer below.
    – Samuel Liew
    Commented Mar 22, 2023 at 2:57
24

Based on a request, I have updated my userscript ReduceClutter (install) to move the "Related questions" module back to it's original position in the sidebar.

This userscript does much more than this though, including stripping tracking from elements like buttons and links, and blocking ads. You can view the source code via the link above.

Some of you may also already have this userscript installed, just click on the install link above to view the diff and update.

Before

enter image description here

After

enter image description here

9
  • 7
    Thanks for the quick update of the script. This way the list doesn't get rubbed under my nose but I can still get the occasional laugh by looking at what the SO algorithm considers related ;)
    – jps
    Commented Mar 22, 2023 at 8:00
  • Any chance "one day" you might correct the ugly Typo/Grammar Horror in the 1st Sentence (in the @description Tag)...? (=> "updates that make[s]") I still have v1.26 of your Script (running on FF55 + GM v4.1, from around 2017, I think, as the Script doesn't contain any Dates/Changelog) and that "Ugliness" was already there and has never been corrected... => in v3.0.2 hopefully...? :wink: (+ Calling everything "new" doesn't really make sense, from a Coding point of view, if Versions and/or Dates are not mentioned... Could check the 'Meta' Thread for each, but that's a bit tedious...)
    – chivracq
    Commented Mar 22, 2023 at 23:29
  • This reminds me of the very popular Windows Vista feature which let you revert back to the previous version. There's got to be something badly wrong with an "upgrade" when early in its life, somebody is releasing a script to undo it.
    – Dawood ibn Kareem
    Commented Mar 23, 2023 at 0:08
  • Oh...!?, that was quick...! 'Reduce Clutter' (with a Space btw) updated to v3.0.2 and the "Grammar Horror" has finally been fixed... I'm impressed...! And it looks like all instances of "new" have disappeared... (Got automatically updated on PM v26.3.3 + GM v1.15.1. (Yep, GM = 'Greasemonkey', and I really mean v1.15.1, ah-ah...!)) // Discovered 'SE Wider Mode' and looks nice, but lots of (vertical) White Space on 'Active Questions' (2cm per Row), on FF110, TM v4.16.6160).
    – chivracq
    Commented Mar 23, 2023 at 8:20
  • @chivracq Not even GM 3???? I can understand not going with GM 4, but why not GM 3 (which, IME, worked well in FF prior to the introduction of WebExtensions)?
    – Makyen Mod
    Commented Mar 23, 2023 at 19:10
  • @Makyen, boah..., why should I...?, GM v1.15 still works perfectly for "my Needs", why should I change/update, ah-ah...!? And PM ('Pale Moon') is not FF, PM26 =~ FF47, => based/forked on FF26, but =~FF47 for Patch/Security-Level. // Check my Profile, I'm a Specialist in Web-Automation, I always use the "best" FCI (for Performance/Reliability-sake) for the Sites I need to automate... (and for some nearly 15 years...) // [1/2]
    – chivracq
    Commented Mar 24, 2023 at 0:54
  • @Makyen I had posted a mini-Compliment under your "fake" Answer to anti-roomba this Thread, (now deleted), I've now been running your 'Roomba Forecaster' Usersccript for a few days (on FF110, TM v4.15.6160, Win10_Pro 22H2), I quite like it... Nice... [2/2]
    – chivracq
    Commented Mar 24, 2023 at 0:56
  • 5
    @chivracq If you're going to go to the effort to write a full paragraph asking for a minor change due to a typo, and referring to it as "Grammar Horror" at that, please at least have the decency to work on your own grammar. "Sentence", "Script", "Dates", "Changelog", "Coding", "Thread", for example, none of those should be capitalized. Change log is also two words. Not to mention the general punctuation and flow made that comment painful to read. At the very least don't be petulant while asking for it to be fixed.
    – Nick is tired
    Commented Mar 24, 2023 at 22:09
  • @Nick, yep you are probably right, I'm not always "perfectly diplomatic", I care for Quality and I didn't know yet but I possibly like "petulating", and yep I like Capitals, ah-ah...!, but just one mini-Mention and #Sam_Liew the Maker/Maintainer of the Script reacted/updated their Script within a few hours, perfectly reacting to my "Bug-Report". nothing wrong with that I would think..., I'm pretty impressed myself how they (#Sam...[]) reacted, I find it super-perfect-professional, (Even if I'm still surprised they never noticed that "Spoink-Spoink" in about 6 years, you know what I mean..)
    – chivracq
    Commented Mar 25, 2023 at 11:20
21

I hope it can be turned off per site.

The posts appearing on Stack Overflow em Português are only "related" by chance, and it's another change our community didn't asked (but we have some asked ones, if someone have some spare time...)

About this:

As part of the experiment, we observed a statistically significant 900%+ increase in clickthrough rate in experiment/variant groups compared to the control group.

Of course. Unrelated links will increase clickthrough rate, as people click and don't find what they want, so they have to click again elsewhere and so on. In terms of finding the content, it's a metric that reflects failure.

I already read that currently it's appearing on other sites by some mistake, but I still think this is a pertinent point if someone decides to apply it network-wide some day.

1
  • 10
    Believe us, it's not better anywhere else.
    – tripleee
    Commented Mar 22, 2023 at 17:26
20

I have nothing to say about this new position that other haven't said yet. But if you insist on keeping this location for Related questions, then at least allow us to minimize that section completely. Now there is expanding box to show more related questions, but there is no way to reduce the view to zero.

There is plenty of room to add additional expander/minimizer next to the title.

2
  • 4
    This could be particularly useful for someone trying to see the question as they write their answer.
    – Ryan M Mod
    Commented Mar 23, 2023 at 17:18
  • 1
    Minimizing to the sidebar is how I'd do it personally.
    – wjandrea
    Commented Mar 23, 2023 at 22:09
19

(or maybe )

The "new answers" notification appears below the related questions, which makes it hard to notice. Please move it above.

Screenshot showing the problem

For completeness, this is what it looks like after clicking the notification.

Screenshots from this question

2
  • Agreed. The answers are always more relevant than the links. What would happen if you click to load that answer, will the related questions remain where they are shown, be shifted below the answer(s) or migrate to the right part of the screen where they should end-up after a page-load since it is now an answered question?
    – Mast
    Commented Jun 28, 2023 at 9:48
  • @Mast The second screenshot shows what happens. Or do you mean what should happen ideally?
    – wjandrea
    Commented Jun 28, 2023 at 15:25
18

When Stack Overflow itself enters the "click count" game, I fear for the future of the platform. It seems there is a big confusion, it's not a social media or anything alike, we're not on YouTube, we are on Stack Overflow!

First things first, if the suggestions in the sidebar were ignored that much, it is certainly not because of its position in the UI.
The reason is simple, the suggestions are completely unrelated to the topic of the question, period.

Therefore, the real problem to solve was not to force people to click on the suggestions by moving the suggestions in the middle of the UI (making the navigation experience even worse by the way).

This is very intrusive in the way the users intend to use the platform and only brings more clutter.

The real problem to solve is to improve the quality of the suggestions.
It's not a matter of UI design, it's about the search algorithm/selection criteria design.

The suggestions were in their exact right place on the sidebar.

9
  • 2
    I mean, if by moving it from location A to location B it gains 900% increased click rate, clearly moving it has affected user engagement with it. I don't think claiming it's old location isn't why it wasn't getting clicks holds water, if moving it clearly changes that.
    – Kevin B
    Commented Mar 23, 2023 at 19:32
  • 2
    @KevinB If I moved some yard decorations directly into the pathway from your front door to your car in the driveway, there's an increased chance you are forced to interact with it, accidentally or not, am I right?
    – Drew Reese
    Commented Mar 23, 2023 at 19:36
  • 1
    You are correct, however, my point is the location is absolutely part of the problem. The right column is filled with garbage, moving garbage out of it and into a more prominent location will increase interaction with that garbage.
    – Kevin B
    Commented Mar 23, 2023 at 19:37
  • And Fareanor's point isn't that the location is what made it garbage, but the fact that it's already just garbage.
    – Drew Reese
    Commented Mar 23, 2023 at 19:38
  • Yes, however, that's not the point of this change, ;) It wasn't an investigation in how to make related questions better, it was an investigation into reducing the bounce rate.
    – Kevin B
    Commented Mar 23, 2023 at 19:39
  • 3
    Is the point then something about polishing a turd? 🤷🏻‍♂️
    – Drew Reese
    Commented Mar 23, 2023 at 19:40
  • What i don't get is why there was such support for the overall initiative, when changes like this is precisely what they were obviously looking for. "In 2022, 82% of our traffic came from organic search. For the most part, these users visit one page and bounce, never exploring further or engaging with the community." literally, all that matters in this whole initiative is reducing bounce rate. that's it.
    – Kevin B
    Commented Mar 23, 2023 at 19:42
  • 7
    Soon: "Click again to see the second half of the top answer!" => +1,000,000% CTR! Success!!
    – Michael come lately
    Commented Mar 23, 2023 at 21:53
  • 'Is the point then something about polishing a turd?' should be the title of most of SO's 'design' announcements
    – pilchard
    Commented Mar 27, 2023 at 13:12
11

I don't think this feature makes sense for the average logged-in user. Anonymous users? Sure, they're literally searching for content. Showing more content when appropriate in that case is useful. If someone from google spots a "related question" they find interesting, there's nothing lost there. For people like me, people who are here for posting answers/helping others and curating content, this list is often just a reminder of low quality posts that got lucky enough to hit HNQ that we can't do anything with. It isn't useful, and it's often even frustrating rather than just being a list of seemingly random questions.

Dial it back to just anonymous users. Not every feature that increases CTR for anonymous users is a good change for all users.

5
  • 3
    It doesn't make sense for anonymous users either, because the recommendations are simply nonsense - at best they're a list of popular questions with similar tags. Also, just being anonymous doesn't mean the users don't have an account, they might simply not be logged in in that context. Personally, I'm logged into SO in my main browser but often use it elsewhere, e.g. in a private browsing window if I don't want stuff to pollute my history, in separate browsers I use for testing webapps when I'm testing and quickly need to look something up, or even on remote systems (e.g. cytrix etc).
    – l4mpi
    Commented Mar 22, 2023 at 15:55
  • 1
    I disagree because not everyone who is clicking those links are doing it because they think the recommendations are useful to the problem they're solving. It's more likely they're just exploring "interesting" questions. It's far more likely for someone who is just a passive user to explore in this way than a more active user.
    – Kevin B
    Commented Mar 22, 2023 at 15:56
  • 7
    Fine, but in that case, it should be namend "popular questions" or whatever, not "related questions". They're simply not related in any reasonable sense of the word.
    – l4mpi
    Commented Mar 22, 2023 at 15:58
  • 1
    I don't think that anonymous vs logged-in is the right distinction here. For one thing, it should certainly be shown to, at minimum, the asker. It would also be useful to suggest possible duplicates to people who might be answering or who could flag as duplicate. Additionally, I am usually logged in when researching a problem - why should I not get related questions if I'm on a page with no answers? The problem, really, is that the suggestions are bad. Either it's useful to both logged-in and logged-out users, or it's useful for neither.
    – Ryan M Mod
    Commented Mar 23, 2023 at 17:22
  • 1
    @RyanM i wouldn't find a list of questions under a question useful even if they were improved. It most certainly wouldn't be useful to all, even if only the best of the best Q&A pairs were somehow selected. It's simply not what I'm here for.
    – Kevin B
    Commented Mar 23, 2023 at 17:25
6

Another example of "these related questions are mostly unrelated to the question".

The question (How to run a *.dat file on Clion) was:

Screen shot of "How to run a *.dat file on Clion"

The list of related questions was:

enter image description here

The top five suggestions are wholly irrelevant. Numbers 6 & 7 do mention Clion, but are not really relevant; number 8 is wholly irrelevant; numbers 9 & 10 could be relevant. So, of the 10 suggestions, 6 are unequivocally irrelevant, 2 are dubiously relevant, and 2 could be pointing in the right direction.

The original question isn't very good. Data files are not usually executable, but that could be because the OP doesn't recognize what they want to happen, or doesn't know how to describe what they want to happen. Nevertheless, the related questions are mostly unrelated.

I think there's some room for improvement in whatever AI is generating these 'related question' suggestions.

2

During the experiment, we observed a statistically significant 900%+ increase in clickthrough rates (CTR) in both experiment groups. Users in the experiment were engaging with related questions at an exponential rate compared to those in the control group where related questions were shown in the sidebar.

I'm not sure what engaging with an exponential rate really means (over time?) but the 900% increase in clickthrough rates in both experiment groups sounds impressive.

Would it be possible to compare click through rates at the end of the experiment in March with rates now? Or did the rate go down again because the novelty effect of the change has worn off.

It's important to check if the observed effects are really that durable. If not, one needs to be more careful when judging the results of the experiments.

-67

As you may have already noticed, we graduated the experiment to display related questions on pages that don't have answers yet. This module will only be shown on Stack Overflow question pages that don't have answers yet. Additionally, we are aware of the module appearing on other Stack Exchange network sites and are working on addressing this. We apologize for the inconvenience caused.

As part of the experiment, we observed a statistically significant 900%+ increase in clickthrough rate in experiment/variant groups compared to the control group.

14
  • 46
    The module is an inconvenience regardless of where in the network it is. Please revert the change
    – Zoe Mod
    Commented Mar 21, 2023 at 19:10
  • 1
    Don't Qt's closed as Duplicate count as having an Answer...? Example (Closed with 3 Dupe Targets, all 3 with Answer(s) of course...!)
    – chivracq
    Commented Mar 21, 2023 at 19:14
  • 29
    observed a statistically significant 900%+ increase - fine, but I hope there's more to it than just a "clickbait". As seen in some of the recent comments, many regular users will now totally block that list and the occasional visitor who asks a question will maybe click on some of the "related" questions and wonder about the purpose, without really seeing any benefit.
    – jps
    Commented Mar 21, 2023 at 19:17
  • 3
    What is different about SO that makes it a valid target for this change and not the other network sites? Surely if its a positive change for SO it'd be a positive change elsewhere?
    – Kevin B
    Commented Mar 21, 2023 at 19:24
  • 30
    Jumping on the bashing the statistics-train here, a high clickthrough rate is great for flexing to the shareholders, but it doesn't measure utility for the end-user. There are plenty of ways in which this is worse, notably by being a waste of space.
    – Zoe Mod
    Commented Mar 21, 2023 at 19:27
  • 1
    @KevinB when we first introduced the initiative we set expectations on MSO that we would be experimenting on Stack Overflow. We haven't set any expectations for the rest of the network. Despite the positive results we saw on Stack Overflow we can't assume it would yield similar results on the rest of the network sites.
    – tanj92 Staff
    Commented Mar 21, 2023 at 19:27
  • 25
    What was the calculus for this change, which no one asked for, over the list of bugs and feature requests that are still outstanding?
    – TylerH
    Commented Mar 21, 2023 at 20:18
  • 31
    It's 900% more annoying having the "related questions" box down there, compared to where it used to be. It's usually useless, now it's still useless and in my face. Sigh.
    – thirtydot
    Commented Mar 21, 2023 at 20:50
  • 21
    @tanj92 funny how neither the initial announcement nor this graduation is featured. Were you hoping nobody would notice and once again hide behind "we received little negative feedback"?
    – Phil
    Commented Mar 21, 2023 at 22:39
  • 6
    Just piling on to the other comments - this change is bad as the "related" questions are in most cases just unrelated random posts, so positioning them more prominently is a bad idea. I noticed it just now under a question closed as a duplicate, where it is especially bad as it's competing for attention with the duplicates list in the close notification. Please rollback ASAP.
    – l4mpi
    Commented Mar 22, 2023 at 12:28
  • 5
    Missed the AB test, sorry, or I would have commented soner. "Related questions" are completely unrelated and a major annoyance. As someone who frequents C++ questions I was unsurprised to see this list show up for a question - these are simply high-vote C++ questions that have nothing to do with the question asked; the "What is the --> operator in C++ Q?" is unrelated to any legitimate question as it is just a confusion about a joke.
    – davidbak
    Commented Mar 22, 2023 at 13:45
  • 9
    Simply put: I hate it. Sorry. I don't care a whit about the click-through rate, and I don't know why you do.
    – John Bollinger
    Commented Mar 22, 2023 at 14:00
  • @JohnBollinger re "I don't care a whit about the click-through rate, and I don't know why you do" - a click-through is an additional page visit, which means an additional opportunity to show ads, so the simplest explanation as to why SO cares is ad dollars.
    – l4mpi
    Commented Mar 22, 2023 at 14:45
  • 6
    And now I've added Related Question to be filtered out by my add block. It is extremely irritating to deal with it below questions. When it was on the right side, it wasn't a problem and could easily be ignored.
    – ChrisMM
    Commented Mar 22, 2023 at 15:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .