72

Over the past two years or so, I have noticed a steady rise in the fraction of new questions - especially Python questions - about some machine-learning problem, wherein the OP is trying to improve an AI model that functions without error, but gives low-quality output. Because it is an AI system operating on a possibly vague task, it may be hard to say objectively that the output is wrong per se, but people will report symptoms such as apparent overfitting or underfitting, attempts at gradient descent that don't "converge" well, unexpectedly low or high "accuracy" self-reported by the model, etc. (Disclaimer: I have at best a very rudimentary understanding of these concepts.)

It appears that in most if not all of these cases, the OP is expecting that the problem should be solved by tweaking some input parameters to the model (most of which is some heavyweight library abstraction), rather than by finding and fixing some problem in the actual logic of the code. Note that I am not talking about questions where there appears to be some logic error in the code unrelated to ML, nor about e.g. ValueErrors raised by Tensorflow when the input data array for a layer has an invalid shape.

Are these questions suitable for Stack Overflow? If not, what closure reason is appropriate?

Can these OPs be pointed somewhere else on the Stack Exchange network?

Can this sort of question be genericized in any useful way? Would it ever be viable to close one question of this sort, as a duplicate of another?

6
  • 8
    meta.stackoverflow.com/questions/404075/…
    – philipxy
    Commented Nov 10, 2023 at 22:51
  • 9
    Well, if the question is about prompt engineering then genai.stackexchange.com should be the right place for such questions, not Stack Overflow. It is ultimately not the programming problem. They can be closed with "not about programming"
    – Dalija Prasnikar Mod
    Commented Nov 11, 2023 at 7:52
  • 3
    One could argue that tweaking parameters of scientific libraries isn't really programming, but the parameters are part of the input data. But maybe others see it differently. If anything I would close as too broad, the number of possible steps for a comprehensive solution are too many. Commented Nov 13, 2023 at 8:22
  • 3
    "Are these questions suitable for Stack Overflow?" How many of them are successfully answered? If they can be answered, there might be a compelling reason to let them stay. Commented Nov 13, 2023 at 8:25
  • 1
    Almost none of these questions are successfully answered, because in the core, they are not programming questions. I deal with this kind of questions every day (in the ML tags), and I vote to close as not a programming question.
    – Dr. Snoopy
    Commented Nov 13, 2023 at 13:02
  • 2
    I don't think that just because certain questions have been answered successfully is any indication of whether they are on topic or not. We might have a hundred questions about how to turn on "dark mode" in Windows 11 (non-programmatically), all answered correctly, but that does not make such questions on topic for Stack Overflow. Commented Nov 13, 2023 at 19:04

4 Answers 4

38

Are these questions suitable for Stack Overflow?

No, but they are unfortunately permitted by our incredibly smart corporate overlords.

If not, what closure reason is appropriate?

When the "lacks understanding" close reason was removed because it hurt peoples' feelings, everyone just started using "too broad" instead. I'd say this is quite appropriate, since it's really not our job to explain the basics of how ML works, to help vampires who aren't interested in understanding that.

6
  • 2
    You're saying such questions are permitted and they should be closed. I don't understand, doesn't "question should be closed" imply "question is not permitted"?
    – cigien
    Commented Nov 13, 2023 at 12:02
  • 11
    @cigien Policy, imposed by SE Inc., permits them. Sanity, imposed by curators, closes them. Just because the company wants this site to become the internet's sewer for help vampires, does not mean we shall allow it.
    – Ian Kemp
    Commented Nov 13, 2023 at 12:43
  • 1
    I'm not aware of any SE inc policy that says such questions are allowed. And anyway, whether something is permitted on the site (as far as topicality is concerned) is not something SE inc decides, it's up to the community to decide. In either case, maybe it would be better to say "... they are unfortunately permitted by the company" to clarify what you mean.
    – cigien
    Commented Nov 13, 2023 at 15:37
  • @cigien done. With tongue firmly in cheek.
    – Ian Kemp
    Commented Nov 13, 2023 at 17:17
  • 4
    @IanKemp Regardless of your viewpoint on SE (the company) or its leadership, can you link to the specific policy imposed by the company that allows those questions? I'm having trouble finding what you're referring to. Thank you!
    – cocomac
    Commented Nov 13, 2023 at 21:11
  • @cocomac I don't think it's really a direct policy, it is more how the company has been making changes over time. Notably they nuked the "too broad" close option which would be a perfect fit for a question like this, and replaced it with a far more ambiguous one which requires mental gymnastics to make it fit a question like this. We all keep pretending that "too broad" still exists (just look at this meta post...), but nah. Not really.
    – Gimby
    Commented Nov 17, 2023 at 9:40
20

Are these questions suitable for Stack Overflow?

No. The reason being the question is truly waaaayyyyy too broad.

Asking about a perceived "bad" performance on a machine learning model is akin to asking about how to make Microsoft Word "better". There is simply too much to consider and too much ambiguity to be remotely answerable as a SO post.

To make the scope of the problem abundantly clear, assuming there's no bugs, here's a list of potential problems off the top of my head about the data only

  1. Is the data "clean"? As in, are there weird data collection errors?
  2. Is the data correctly scaled? Or more generally, is the preprocessing sensible?
  3. Is the data distribution the same in training and testing?
  4. Is the amount of data sufficient?
  5. Is the data even meaningful? As in, are there any correlations in the predictor and outcome at all?

To make matters worse, these questions are not answerable without looking at the data itself and understanding the context from which the data is collected. However, none of this can be included in the question.

Magnify the list of questions five-fold, and perhaps you'll get somewhere close to the minimum number of factors required to answer definitively why some model has "bad" performance.

6

It depends.

  • If the question is asking about how to improve the quality of an AI model using a specific functionality found in some library in some programming language, I think it's no different from any other questions about data analytics libraries in languages such as Python, R, MATLAB, etc., so are in most cases on-topic.
  • Questions about gradient descent not converging well are often implicitly asking about an implementation of a loop which depending on the question could be an X-Y problem, hence off-topic, but generally these are on-topic.
  • On the other hand, questions about overfitting, underfitting, models to use/evaluation metric, etc. are off-topic IMO, because they are implicitly asking about theory which should probably be posted on other SE sites such as Cross Validated, Data Science, Mathematics, etc.

Would it ever be viable to close one question of this sort, as a duplicate of another?

Just like any question on Stack Overflow, I feel like most of these questions can be closed as a duplicate of another, but there are a couple of problems:
a) it might be difficult to set certain questions as canonicals
b) the question askers may not immediately realize that a solution in another question solves their question

1
  • 1
    The problem of canonicals can be solved by creating them as self-answer questions. (When I do this, I make it clear that this is intended to be a canonical to discourage people from down-voting and dup closing them.)
    – Stephen C
    Commented Nov 13, 2023 at 6:23
-20

Stack Exchange's primary purpose is to build a Q&A archive, and its secondary purpose is to connect people who need help with experts who can help them.

There isn't any question that these kinds of questions fit into the second category, but in order to fulfill the first the problem or solution needs to be described in generic enough terms that 3rd parties will be able to get some value out of it.

14
  • I agree, but I don't understand how that is connected to this question. Does it mean these questions are a good fit? Commented Nov 13, 2023 at 8:24
  • 2
    @NoDataDumpNoContribution they are certainly valuable to help the asker, and if either the problem or solution is generic enough, they help 3rd parties. If the problem and solution both only apply to the exact situation of the asker without any possibility of helping anyone else, then no it's not a good fit. Commented Nov 13, 2023 at 8:31
  • 1
    (1/2) A question may only help the asker and be on-topic for the site, for example, questions regarding obsolete libraries that very few people use today. However, the way I understand it, the main concern of this meta post is that such questions don't have an actual problem with the code itself, and are primarily asking about how to improve their model's training by updating the values of arguments passed to them. Such questions discuss problems about the data being passed to the training-concerned functions and not the code of those functions themselves. Commented Nov 13, 2023 at 8:51
  • 1
    (2/2) The other type of questions this post is referring to are related to ML theory, or statistics, off-topic here as well. Commented Nov 13, 2023 at 8:53
  • (3/2) (coz why not), I think that the answer from Ian Kemp describes the current state well and a possible action. Cottontail's answer is very descriptive and sums up my thoughts as well regarding the different scenarios. Commented Nov 13, 2023 at 9:00
  • 1
    You are incorrect about the second category, asking a non-programming question in a programming site, means you are not actually targeting the right experts, many SO answerers are not ML experts that can solve your problem, and in many cases it would require experimentation that goes far beyond what is acceptable in SO.
    – Dr. Snoopy
    Commented Nov 13, 2023 at 13:05
  • @Dr.Snoopy And yet many are, it's a simple question of tagging. To say that writing tensor flows in python is never something programmers would know about doesn't grok! Commented Nov 14, 2023 at 9:41
  • Writing TensorFlow or PyTorch code is very different from ML theory and network design, that is your core misconception.
    – Dr. Snoopy
    Commented Nov 14, 2023 at 10:39
  • @Dr.Snoopy the question is, are there people on the site who can answer the question. The answer to that is yes. If you are writing pytorch code you probably can help someone else with their pytorch code, even if the question is quasi-domain specific (albeit an extremely common domain for programmers, practically all programmers work with ai these days). Commented Nov 15, 2023 at 2:22
  • No, same misconception again, programmers use "AI APIs", they do not know the specifics of how ML/AI models actually work. There are other sites like ai.stackexchange.com where there is an actual community of AI experts, not in Stack Overflow.
    – Dr. Snoopy
    Commented Nov 15, 2023 at 6:02
  • @Dr.Snoopy Whether or not other sites could also be appropriate is immaterial. A games programing question can be asked here or the games stack, for example. Commented Nov 15, 2023 at 7:35
  • But not about AI and model development, there is no intersection in that case. Again you extrapolate incorrectly.
    – Dr. Snoopy
    Commented Nov 15, 2023 at 7:46
  • @Dr.Snoopy There definitely is an intersection between AI dev and programming. Kind of unbelievable to say people using tensorflow and pytorch aren't programmers. Commented Nov 15, 2023 at 8:01
  • Well that is not what I said, here we are talking about tweaking models, not about general AI development. You need to focus on the actual issue under discussion.
    – Dr. Snoopy
    Commented Nov 15, 2023 at 10:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .