People who pretend that these tools have bee developed without human oversight (or would be used without it), are merely showing their ignorance, and their unsuitability for the critic cause. I looked into the study that revealed the 9% figure at the time, and the methodology is as robust as you would expect of actual research.
Of course journalists are going to report such finding uncritically if there's no reason to doubt it - they're journalists, not researchers. It's their job. Anyone who doubts the research, can publish their own study using whatever methodologies they think are better - all the underlying data that study used is freely available, as is their methodology. To date, nobody has, as far as I have seen.
Anyone who doubts the findings of the research, needs to be honest - do they even understand it? If these people can't even demonstrate that, why should they even be listened to? The bias to women for example - it either comes a bias in the training set, or more likely the humans asked to rate the seriousness of personal attacks to train the model. AI is not a traditional algorithm, you can't just add a line that says 'if woman add 10%' (well, you can, but it would be an obvious add-on).
People who think they are proving something by rick-rolling the tool, these retards clearly can't even read.....
The limits of this research include that it only looked at egregious and easily identifiable personal attacks. ..... The model does little for other forms of harassment on Wikipedia; for example, it is not very good at identifying threats.
Wikipediocracy is full of absolute steamers like that. Pointing out their deficiencies, perfectly politely, is why I was banned. I'm still waiting for Cas Liber to find the balls to actually demonstrate I somehow don't know what I'm talking about. He said it. The forum management spared him his blushes, as is their apparent job - protecting shitheel Wikipedians, whose anus is often their mouth hole.
When the worst you can say about the tool is it might under-estimate the number of outright personal attacks, and can't detect all the other things that account for why the apparent level of harassment on Wikipedia is so high based on user surveys, then you need to have a serious word with yourself. Are you a serious critic, or merely a reflexive defender of scum like Eric Corbett and their inalienable rights to be an ashole? We know what Cas is, because he is shameless and readily admits it (and even if he didn't, there is enough evidence to prove that is what he is).
No serious critic with a ton of experience watching the Wikipedians and their failed system of governance, doubts the conclusion of that study....
the majority of personal attacks on Wikipedia are not the result of a few malicious users, nor primarily the consequence of allowing anonymous contributions from unregistered users.
The toxic culture is embedded in the regular user base. People like Fram, and the hundreds prepared to try and deny he has done anything wrong, or at least nothing serious.
If you're gonna do deep dive journalism of Wikipedia, you need to demonstrate you can actually deep dive. Take this......
Criticism of the Detox tool had emerged after it was brought up in connection with the recent banning of a veteran administrator by the Wikimedia Foundation. Users suggested some form of automated system may have been used to examine the conduct of the administrator, though the Foundation has denied the allegation.
Sounds spooky, right? So what happened. Well, FeydHuxtable made a very long post at 12:55, 22 June 2019 where he basically just speculated based on nothing but the fact research like this has been done, to build a narrative where T&S has supposedly expended thousands of dollars and millions of computing cycles to identify the the top ten most likely victims of Fram's harassment, contacted them and built their case on their testimony. This is a markedly different allegation to what is implied by Breitbart on that paragraph, or their linking to the rick-rolling of Detox by Seraphimblade at 15:51, 22 June 2019.
Now, maybe people are stupid and might assume FeydHuxtable and Seraphimblade were just concerned citizens, people whose concerns the Foundation should quite rightly be answering. But of, course this is Wikipedia. In reality, in the pantheon of thories as to why Fram might have been banned, these two are on the extreme fringe. These people genuinely believe Fram did nothing wrong, that literally all he is guilty of, is enforcing policy. They are explicit, what Fram did is covered by
what harassment is not.Now, I'm not suggesting the WMF hasn't used serious technology to study Fram, but if they have, the denial is difficult to explain. Perhaps it is significant that they only appear to have denied using this tool? Everything we know about these sort of tools, tells us that they can assist such investigations, and for all their limitations, they simply wouldn't be identifying any victims of Fram's harassment if all he was doing was tracking their contributions to enforce policy, and those who say they do not, are saying it from a position of ignorance.
The Fram controversy has revealed what all these researchers probably realize pretty quickly - the Wikipedians in general are a very unethical bunch, not remotely geared to dispassionate analysis of anything, least of all their own culture. Harassment is a nuanced issue, it requires the ability to see things from multiple perspectives. Just as Fram's victims feelings can't simply be used to convict him, his absolute certainty in his own mind that what he was doing was a good faith attempt to enforce policy, cannot also be taken as read. Policy says don't ignore the concerns of others. That is just one example of how you can quite easily find Fram guilty, his victim's experience having been closer to the truth.