11
\$\begingroup\$

With the increase in popularity of machine learning and its libraries, we are also getting more machine learning questions.

Machine Learning is a tricky subject as the concept of whether or not the code "works" is even more grey than usual.

Which kind of Machine Learning questions are on-topic here?

  • "My Machine Learning is learning too slow"
  • "My Machine Learning is not learning at all"
  • "The (training/cross-validation/test error) in my Machine Learning is too big"
  • "My Machine Learning is running too slow"
  • "My Machine Learning currently has an accuracy of x%, which is pretty good. Can it go even higher"
  • (other questions I haven't thought of)
\$\endgroup\$
2
12
\$\begingroup\$

I'll replace the "Machine Learning" parts of your question with "code" to keep it simple. After all, that's what it is.

"My code is learning too slow"

But it works, right? If that's the only issue, it's definitely on-topic. Classic case of a question.

"My code is not learning at all"

The entire purpose of ML code is to learn. If it doesn't learn at all, it's broken. Broken code has no place here.

"The (training/cross-validation/test error) in my code is too big"

That's an accuracy problem and where the trouble starts. How good is good enough? Speech recognition with an accuracy of 5%, that would be broken. After all, with such an accuracy it can't be working as intended. But where do we draw the line?

If the question is phrased as "The accuracy is not good enough" it reads an awful lot like questions where not all test cases are completed successfully. Which means the code is broken and not ready for review.

Another grey area would be questions saying "I currently get an accuracy of 70%, which is pretty good. Can it go even higher?".

So I guess it partly depends on how it's phrased. And I don't like that one bit. This is the major question we should be focussing on, I think.

"My Machine Learning is running too slow"

Also a question. No problem.

\$\endgroup\$
5
  • \$\begingroup\$ I added your "I currently get an accuracy of 70%, which is pretty good. Can it go even higher?" to the question. I also added a link to a related topic from a few years back. \$\endgroup\$ – Simon Forsberg Apr 14 '18 at 15:56
  • \$\begingroup\$ "My Machine Learning is running too slow" may often be an algorithm choice, implementation detail or problem-framing issue in ML. I don't know if those would be on-topic here (I've hopped over from Data Science)? \$\endgroup\$ – Neil Slater Apr 15 '18 at 8:26
  • \$\begingroup\$ @NeilSlater Pointing out the flaws in an algorithm can be part of a review. We even have the algorithm tag. \$\endgroup\$ – Mast Apr 15 '18 at 8:56
  • \$\begingroup\$ We could set arbitrary limits for accuracy, anything below 45% accuracy is broken 45% or greater needs improvement. \$\endgroup\$ – pacmaninbw Jan 16 at 15:38
  • \$\begingroup\$ @pacmaninbw I don't think that's a good idea. 'State of the art' for a problem could be 20%, where average is 15%. The user could get 17.5% and wants to know how to make the code better. It's not broken cause it's better then average and there is clearly room to make it state of the art. Now if the question was 5% instead of 17.5% I think we can agree that would be off-topic. We've in the past said arbritrary limits would only summount to a worse problem then what we have at the moment. \$\endgroup\$ – Peilonrayz Jan 16 at 18:46
10
\$\begingroup\$

I think that the nature of the question itself lends to several options. Here are my thoughts based on Simon's examples.

"My Machine Learning is learning too slow"

These would seem to fall into Code Review's realm for performance improvements, if there is a performance issue indeed, but for algorithm issues it may be better on Data Science's performance tag.

"My Machine Learning is not learning at all"

These may be viable for Stack Overflow, if they can provide an MCVE.

"The (training/cross-validation/test error) in my Machine Learning is too big"

These seem to be algorithm issues that very well may be on-topic on Data Science, although we should keep in mind their What types of questions should I avoid asking? page before making recommendations to post there.

"My Machine Learning is running too slow"

These would seem to fall into Code Review's realm for performance improvements. Clarifications on memory/CPU/etc. usage may be useful to narrow down where the issues are.

"My Machine Learning currently has an accuracy of x%, which is pretty good. Can it go even higher"

These may be better fit on Cross Validated, if the issue is of statistical nature, rather than computer code nature.

\$\endgroup\$
-3
\$\begingroup\$

In defence of "My Machine Learning currently has an accuracy of x%, which is pretty good. Can it go even higher", let's take a hypothetical scenario.

Problem: OP has a well-written code for a machine learning task that uses proper syntax and runs optimally fast BUT his model doesn't fit the data as well as they expected.

  • Phrase 1: "I have a model that gives me x% accuracy which is great but can it go higher?"

  • Phrase 2: "I have a model that gives me x% accuracy which is bad, how can I improve it?"

As per my understanding, questions with phrase 1 are encouraged and they should be! In my years of experience of reviewing code for ML, learning capability, loss reduction, overfitting, underfitting, generalizability of model etc are extremely important parts to a review of an ML related code. Without these aspects, we are only tackling syntax improvements, speed improvements and other production-ready aspects for the code pipeline instead of actually reviewing the code for Machine learning.

Phrase 2 will clearly be considered as off-topic here based on the sentiment and comments of other users on CR and SO, that I have seen and that begs the question - is it just a matter of how I phrase my question?


Let's say tomorrow we decide that questions/problems of this type are strictly off-topic on CR. (These already are off-topic on SO). Where does a guy with such a question go then? Clearly, they need their code reviewed and it's not specific enough question for SO either. In this case, are we going to decide that these questions are not to be entertained irrespective of the fact that this is an important and necessary part of code reviews that are currently done in the industry?

\$\endgroup\$
2
  • 3
    \$\begingroup\$ "Where does a guy with such a question go then?" Honestly, we can never take this into consideration because it's not healthy to the site's scope. After all, why should we take it instead of Datascience, CrossValidated or any of the other Software sites that deal with ML? \$\endgroup\$ – Mast Jan 16 at 12:46
  • 2
    \$\begingroup\$ The first problem is something we've just grown to accept. Yes people can lie, and I'm sure people in the past have. But how would you solve it? Isn't the solution just to determine if the author is lying with perfect accuracy? If our best scientists haven't made a perfect lie detector, and the legal systems would pay through the nose for that, then what hope do we have? A solution based on 'working' would go against our rules - "works to the best of the author's knowledge." \$\endgroup\$ – Peilonrayz Jan 16 at 19:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .