19
\$\begingroup\$

I'm aware of When are Machine Learning questions on-topic?, but I think maybe there's room for discussion regarding the tag.

I've made a search on the ML tag (because I do not have access to the SE Query... thing) and figured that there are actually 66% of unanswered questions tagged with . (Maybe my search wasn't good though)

I think that, due to the nature of Machine Learning, it is very hard to make a valuable review regarding performance/accuracy without having :

  1. Access to the data
  2. Time to fiddle with it
  3. Experience in the Machine Learning field

That is without considering the fact that hyperparameters, which are often then key to great performances and can greatly affect speed, are pretty much chosen by trying different values.

I think that machine learning related questions will be more and more frequent and I'm wondering if they are really fitting here or if they should be posted to one of : ArtificialIntelligenceSE, DataScienceSE, CrossValidatedSE, etc(SE).

| |
\$\endgroup\$
  • 3
    \$\begingroup\$ Sidenote: The "SE Query... thing" can be found at data.stackexchange.com \$\endgroup\$ – Vogel612♦ Feb 8 '19 at 14:06
  • 1
    \$\begingroup\$ @Vogel612 well thanks! I thought it was reserved for the 10k+ users. I've validated my numbers \$\endgroup\$ – IEatBagels Feb 8 '19 at 14:10
  • 1
    \$\begingroup\$ "I think that, due to the nature of Machine Learning, it is very hard to make a valuable review" absolutely. So if other sites can handle such questions better, we should ask them what they can do with it. \$\endgroup\$ – Mast Feb 8 '19 at 14:23
  • 3
    \$\begingroup\$ Just because it's hard to answer the questions here doesn't mean that they are a bad fit for the site itself. \$\endgroup\$ – Simon Forsberg Feb 8 '19 at 15:54
  • \$\begingroup\$ @SimonForsberg No, but they may be a better fit elsewhere. If they are, everybody would win by pointing them into the right direction. \$\endgroup\$ – Mast Feb 8 '19 at 18:34
  • 3
    \$\begingroup\$ @Mast As long as you don't vote to close them with that reason, that's fine. Just be sure you understand the scope of the target site before you redirect people there. \$\endgroup\$ – Simon Forsberg Feb 8 '19 at 22:26
  • 1
    \$\begingroup\$ @SimonForsberg Of-course, I wasn't suggesting that. Do we have regulars among us that happen to know a decent bit about the scope of the sites mentioned? They may be able to shed some light in the answers. \$\endgroup\$ – Mast Feb 9 '19 at 12:41
  • 1
    \$\begingroup\$ But should we have some of them as a target for closing? For example the question codereview.stackexchange.com/questions/213806/… would probably be on-topic at Data Science. \$\endgroup\$ – Graipher Feb 19 '19 at 15:57
4
\$\begingroup\$

I think it depends entirely on what is being reviewed.

Reviewing the choices made for fitting the model (choice of parameters / transformations / type of model) are less about the code and more about machine learning as a whole. I think these kinds of questions would be better suited on the suggested alternatives, because it's not as much reviewing the code, but the idea behind it.

On the other hand, working with large volumes of data means that writing good code can make a big difference. Looping over the data unnecessarily, huge joins filling up RAM, those are things that can be reviewed by looking at the code itself.

Languages like Python make it really easy to start working with ML without having much programming experience. That seems to me like a perfect oppurtunity for a review :)

| |
\$\endgroup\$
-2
\$\begingroup\$

TL;DR

Overall I find the reasoning to be fairly poor. All the arguments for the tag to go can be attributed to most questions regardless of tag. And any reasons related directly to the tag itself can be attributed to other tags too.

Overall if we ban ML then we should ban things like NumPy or JavaScript for the same reasons. It just doesn't make sense to me.

Furthermore I think the only point that hold ground, should be resolved with higher quality controls. However I don't think this should be the post to get into that.

Answer

Answer rate shouldn't be used to verify if something is on or off topic. This is a slippery slope.

At the time of this answer your statistic that 66% unanswered questions for ML is way off. Given that it's been 8 months, that may be why. A quick way to see the unanswered rate of tags it to go on the 'top users' part of the tags, here's the machine learning top users. Here we can see that the tag has 34.3% unanswered.

Like all statistics you can make things sound better or worse than they actually are.

Over the same time period Machine Learning has had a 0% unanswered rate, where JavaScript is suffering from a whopping 55.6% unanswered rate.

Yeah, these are both over the last 7 days.

More sensibly this isn't a good metric. I used to look at the top users page to see who's close to the 1k gold Python badge. And all the time I saw ~30% unanswered rate for Python consistently for the past 7 and 30 days. The number has gone down a bit right now, but it's still around 20-30%. If this is a forecast on how the all-time number is going to go then at what point do we have to say "no more Python, we just can't handle it."

Whilst this may sound fairly ridiculous right now, give it 20 years and the Python tag may actually have a 30% all-time unanswered rate.

I think that, due to the nature of Machine Learning, it is very hard to make a valuable review regarding performance/accuracy without having :

  1. Access to the data
  2. Time to fiddle with it
  3. Experience in the Machine Learning field

I feel these are particularly poor reasons to reject ML as they, within reason, affect the entire site anyway.

  1. It's always useful to have the I/O of a program. With ML it's probably more important, but that doesn't mean the question isn't answerable. Maybe not in the way the OP wants.

    Rather than banning a set of questions that tend to have a higher reliance on providing I/O, why don't we require more information than we previously have for these questions, or all questions in general. This would have the benefit of having higher quality questions, but we'd be closing more questions.

    I don't think this is the question or answer to go into this, but I think it'd be the better option.

  2. This seems like a pretty strange reason to not allow a group of questions. We got our question character limit increased to 64k or something so we can have larger programs. These larger programs take more time.

    I don't think this is a reasonable reason to ban ML. We have had people complain on meta that posting answers to Code Review is long before. That's just how the site is...

  3. To me this as a reason is fairly mind boggling. We have very few NumPy professionals here, but does that make NumPy off-topic here?

    I was hanging around the Python Stack Overflow room for a bit, and a couple of times recommended people to come here if they want their Python code reviewed. But always stated that NumPy questions probably wouldn't do well, as we don't have NumPy pros.

    I don't think NumPy should go, it's massive over on SO. We just haven't got traction.

I think that machine learning related questions will be more and more frequent and I'm wondering if they are really fitting here or if they should be posted to one of : ArtificialIntelligenceSE, DataScienceSE, CrossValidatedSE, etc(SE).

I'm a little disappointed that a fairly high-ranking CR user would utter such words. We have to fend off this mentality from SO, let's not fester the mentality internally too.

Whilst other sites may be better for some questions, without looking, I highly doubt they will allow code reviews.

| |
\$\endgroup\$
  • \$\begingroup\$ If I remember correctly, at the time I posted this question I saw a lot of questions that looked like "The performance of my model is poor, why?". I guess in retrospect I might have formulated this better. I think the real problem might be more with tracing the line as to what defines working code regarding machine learning. Say, the model predicts with a 25% accuracy. It runs, it gives a result, but can we say the code works? What should we do in these situations? I think there are some good machine learning questions, but I also think that given the "black box-ness" of some... \$\endgroup\$ – IEatBagels Nov 14 '19 at 0:55
  • \$\begingroup\$ algorithms I find myself confused as to how we should define working code and what we can do as a review without passing 5 hours training a model and observing curves to figure out the learning rate should be 0.005 instead of 0.0001. I hope you understand better what I think I was trying to convey (Although, as I said, I don't think I had the crosseye on the problem at the time). \$\endgroup\$ – IEatBagels Nov 14 '19 at 0:57
  • \$\begingroup\$ @IEatBagels Ah yes, that's completely different to what I read. I think we have a meta stating something about that, but I can't find it :( \$\endgroup\$ – Peilonrayz Nov 14 '19 at 1:04
  • \$\begingroup\$ @IEatBagels Yes, the main problem was we had a lot of questions stating they wanted to improve performance, while they meant accuracy. Some of those questions had an accuracy of 30% or less. Simply put, with an accuracy that low the code doesn't work yet and we should consider it broken. Thing is, where do you draw the line? Unit-testing ML code isn't a thing, I think. \$\endgroup\$ – Mast Nov 14 '19 at 12:21
  • \$\begingroup\$ @Mast The thing is, 30% might be state of the art for a problem and obviously broken code for another. \$\endgroup\$ – IEatBagels Nov 14 '19 at 13:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .