"Toxic comments are associated with reduced activity of volunteer editors on Wikipedia"

[ericbarbour]

no shit omg what a surprise i never imagined

https://academic.oup.com/pnasnexus/arti ... ogin=false

Notice that "toxic comment" is never defined or specified. Is this another trial balloon for open censorship of every WP talkpage? It has been considered before over and over (BADSITES being the most notorious).

Soon it will be "Wikipedia is NOTCENSORED, however...."

[Bbb23sucks]

"Free speech exemptions"

[ChaosMeRee]

A mixed bag.

It does seem to be bringing something new to the table....

It is worth noting, however, that it has already been established that toxicity on its own could lead users to stop contributing either temporarily or permanently, as this is what editors themselves report in surveys (24). Our study complements such studies by providing an estimate of the potential effects while also being performed on a scale that is not achievable by survey methods.

Their definition of toxicity seems reasonable and well founded.....

In this paper, we use a model from the Perspective API (68) to identify toxic comments.

....

Each comment is labeled by 3–10 crowdsourced raters. Perspective models provide scores for several different attributes, see Supplementary Table S2 for the list of attributes and their definitions, see Supplementary Table S2 for examples of toxic comments, and see Supplementary Table S3 for the AUC (Area Under the Curve) scores for those languages and attributes that were used in this paper.

We define a toxic comment as a comment that has a score of at least 0.8 on any of the six dimensions provided by Perspective API. The score means that on average 8 out of 10 raters would mark it as toxic.

The six dimensions are Toxicity, Severe Toxicity, Identify Attack, Insult, Profanity, Threat.

The examples seem highly manufactured, but it has to be said, setting the bar at 0.8 seems very generous. There are examples rated at 0.62 and 0.73 that are pretty severe incivility, throwing around terms like "hypocrite" and "troll", and even naked threats.

That probably highlights one major flaw of the study. As we know, in terms of perceived (and by rule) levels of incivility, whether terms like "troll" come with diffs, matters greatly, and indeed you can get away with pretty severe threats too, if you bring the diffs.

it's main purpose appears to be to highlight that it is the community themselves who are preventing toxicity from being handled, by repeatedly rejecting sound proposals that speak to evidence of harm being done to Wikipedia, so the immediate impact will likely be nil.

What the scientists don't seem to get, is that on Wikipedia, certainly when it comes down to voting on major policy changes, voting based on how you feel is considered just as valid as bringing a study.

You can see why they would be confused, since WP:CONSENSUS clearly states that evidence free opinion votes should be given the least amount of weight.

[ChaosMeRee]

In case there was any doubt this is a pro-Wikipedia study by pro-Wikipedia academics....

https://theconversation.com/wikipedias- ... you-218517

Every day, millions of people worldwide use it for quick fact-checks or in-depth research

And what percentage of those have read and understood the Wikipedia disclaimer?

Anyone using Wikipedia for a quick fact check is probably a certifiable moron. "Quick" if it is to mean "Quicker than Googling it yourself" most definitely indicates the disclaimer was completely ignored.

Anyone using Wikipedia for "in-depth research", if they are doing that the way Wikipedia says they should*, is probably spending more time than someone who completely ignores Wikipedia's existence in their pursuit of a specific area of knowledge.

* - check every source given exists and verifies the text (including of course paying for any subscriptions required and obtaining offline sources), check those sources are not products of citogenesis, and if the article is not "Featured" quality, do your own literature review to reassure yourself the Wikipedia article is not biased or a less than comprehensive treatment of the subject.

It is of course never a good look when academics make bold claims about the usefulness of Wikipedia, don't seem to even know the first thing about Wikipedia....

Wikipedia relies solely on the work of volunteers: no one is paid for writing or editing content.

Lolwut? On page 2 of any Unbiased Introduction to Wikipedia For Serious Researchers would be the following paragraphs.....

Wikipedia does allow paid editing. And even though this is only permitted with full public disclosure, neither the Foundation or the volunteers have made any effort at all to determine what percentage of editors are paid and what percentage of Wikipedia content has been paid for. There is no central register of legitimate paid editors nor a database of articles featuring paid content. There is not even a standardised method of disclosure. And like everything on Wikipedia, hence the disclaimer, it is down to volunteers to do even the basics and alert readers to the fact they are reading an article that features paid content. The fact that Wikipedia allows paid editing is not even mentioned in the Wikipedia disclaimer.

An unknown amount of disclosed paid editing (including payment in kind) is done with the full and complete knowledge of and endorsement by the legal owners of Wikipedia. Not just its existence, but its aims and objectives. Various different contractual arrangements exist between Wikipedia or their affiliates and these paid editors. There is even a mechanism for peer to peer arrangements. They are typically not called paid editors, presumably due to the negative connotations, and are instead given pseudo academic titles, such as "Wikipedian-In-Residence", in a similar way to Wikipedia calls volunteers "editors". Peer to peer payments are called "rewards".

Even though Wikipedia has a clear policy on paid editing, it is an open secret that undisclosed paid editing (UPE) on Wikipedia is RIFE (hence any time the volunteers have to devote to tracking paid editing is towards undisclosed agents). And despite this being clearly one of the biggest threats to Wikipedia's alleged integrity as a source of unbiased reliable information, there have been zero efforts by the Wikipedia Foundation or indeed its volunteer editors (who admittedly would lack the resources to do a proper job of it) to study the phenomena of UPE with an academic rigour. It is currently impossible to even give a vague but evidence based estimate of how likely it is that a Wikipedia editor is an undeclared paid editor or that a specific piece of Wikipedia content is the result of UPE.

Even though Wikipedia content is possibly (see above) mostly viewed as the product of volunteers doing it out of only the goodness of their hearts, it is an undisputed fact that contrary to the founding principles of the so-called Wikipedia movement, it is now virtually impossible for them to create, distribute and facilitate the re-use of this content without the assistance and cooperation of the paid employees of the Wikimedia Foundation or use of its assets. In extremis, it would take millions of dollars and several years to successfully "fork" Wikipedia, even though the content is legally free to copy. Any study of Wikipedia's "community" of " volunteers" that neglects to account for these powerful levers of influence that rest in the hands of paid actors, is inherently flawed.

[ericbarbour]

And what percentage of those have read and understood the Wikipedia disclaimer?

ZERO of course. Even the journalists, who are putting their jobs on the line by blindly copying "factoids" from WP, have never read the disclaimers nor any of the warnings.

[ChaosMeRee]

Wikipedia is a would-be meritocracy that hasn’t come to agreement about exactly what constitutes merit.

Au contraire, mon cherie.

There is wide and strong agreement.

Merit = Belief Wikipedia Is A Good Thing

This explains the phenomena hinted at later in the thread....

* Hemiauchenia, an absolute disphit non-expert, but deeply committed to the idea Wikipedia Is A Good Thing (seven years, nearly 50,000 edits).

Totally unblockable, even when caught red handed at WP:AN directly personally attacking all student editors and their Wiki-Edu mentors. Not blockable even when seen running to Wikipediocracy to WP:CANVASS for support (all such behaviour is seen because at any one time there are a billion Wikipedia Administrators and Arbitrators viewing and even posting at Wikipediocracy).

* Professor X, published expert in Topic X, sporadically edits Wikipedia In Topic X, but isn't a committed Wikipedian because they disagree that Wikipedia is anything but a nuisance (they probably also have a career and a life and don't need Wikipedia to give their lives significance the way a no-mark like Hemiaucnenia clearly does)

Totally blockable in a scenario where he ends up in an editing dispute with Hemiachenia and loses his cool because arguing with a complete twat is a waste of anyone's time, especially a Professor.

Both editors are clearly toxic. Only one is an Unblockable.

The deciding factor is Wikipedia's definition of merit.

You can't do shit on Wikipedia if you disagree with this definition of merit. It lives and spreads freely, even among the fifteen High Priests of Wikipedia who get to choose who is and is not a good Administrator (the people tasked with ejecting toxic editors!)

Scottywong was despatched because he was an outlier. He had the incorrect definition, and dared to apply it to a dirty little weasel like Eric Corbett.

These papers are proving that to be a monumentally stupid decision. Even the most ardent defenders of Eric Corbett can probably see that keeping him around at all costs, didn't mitigate the costs detailed in this paper.

From 'Examples of comments with different toxicity scores' in the paper:

I wanted to let you know that I just tagged [[article name]] for deletion, because it seems to be vandalism or a hoax. If you feel that the article shouldn’t be deleted and want more time to work on it, you can contest this deletion, but please don’t remove the speedy deletion tag from the top.

This scores 0.21 on the toxicity scale. Low, but still apparently 'toxic'. In order to prevent such toxicity, should Wikipedia stop deleting hoax articles and vandalism?

The paper seems to be perpetuating the usual 'anyone can edit' BS, assuming that the loss of any 'contributor' is by definition a net negative, and working on the premise that it is somehow possible to create something aspiring to be an encyclopaedia without ever upsetting anyone, including those who are actively seeking to degrade it.

HEY DICKHEAD.

Read the example carefully. Why would anyone who was genuinely committed to CIVIL even write such a thing?

If an experienced editor is sure something is vandalism or a hoax, they would not invite the person who created it to "work on it".

If they believed it was a hoax and the editor was an unwitting victim, they would not invite them to "work on it".

If they're unsure whether it is a hoax, they would not tag it for speedy deletion (it is not an Administrator's job to research hoaxes).

If they're unsure whether it is vandalism, they're clearly incompetent and should have been blocked a long time ago.

The AI producing these toxicity ratings was trained in part on Wikipedia. The way that post reads, is an asshole editor being superficially polite while just wandering around slapping people with pro-forma templates that may or may not even be applicable, but place all the burden of figuring out the precise reasons why they were just bitch-slapped and what they need to do now, on the victim. A trainer would see that, and rate it accordingly.

Believe it or not, there are people or who view that kind of arrogant my time is more valuable than yours approach to Wikipedia editing, to be hostile. The asshole editors of Wikipedia don't like it and many go to Wikipediocracy to cry about it, but fuck them, they are pieces of shit.

Not the worst crime, hence the low rating. But given you couldn't see the issue and you are a well known asshole, I'd say the AI is doing its job here.

I'd say that on Wikipediocracy if I could, but as we all know, Jake likes to protect and encourage his special brood of assholes. I'm not allowed to play at all.

[Bbb23sucks]

The paper's first author has joined WPO and has commented: https://wikipediocracy.com/forum/viewto ... a3#p338600

[ChaosMeRee]

The paper's first author has joined WPO and has commented: https://wikipediocracy.com/forum/viewto ... a3#p338600

And it is going exactly how one would imagine.

Wikipediocrats can't do science. Like their seed population. But hey, they give it a good go.....

You should start examining some of the elements of your hypothesis more closely.

And to be clear, you don't think that applies to someone claiming "Wikipedia is largely built out for major articles and is in more of a sustaining mode" or "early offerings of primitive AI, along the lines of ChatGPT, can replace most of the editing functions being done manually"?

Is Wikipedia in "sustainability mode" out of choice, or due to an acute labour shortage? *checks quality ratings of Level 1 and 2 Vital articles*. Oh no, wait, I think I got the answer! Good old close examination wins again. Hard to argue Wikipedia is contentedly sustaining itself by choice when only 11 of its 150 most important articles have reached their own self certified standard of being their "best work".

As I suspected, this tool fails the Thesaurus attack.

"Eat a bag of dicks and die in a fire." <-- That's a problem.
"Consume a satchel of Richards and expire in flames." <-- Okie dokie

Hmmm. Further research most definitely not required. The witty retort of subject B is not a problem because it would naturally help retain users, eager to bask in the superior intellect of their enemy turned mentor.

We call this the Eric Corbet phenomenon down at the lab. The Vigilant phenomenon, the feeling of dread, depression and disgust as you observe a tired old dog pissing itself while it ever so gently gums at your leg, is quite a different thing entirely.

Perhaps a future area of study might be to look at the origin of the toxic comments. My guess is that newer users are more likely to leave toxic comments because they are unaware of the conduct expectations, have less time invested, and have not yet been warned/blocked for their behaviour

Your "guess" is coincidentally an extremely common view held by the Wikipedians. It very successfully permeated out into the world as Wikipedia went mainstream, because it is quite persuasive to the layman and those who have never spent much time in the guts of Wikipedia. And of course, it being a commonly held view suited Wikipedians, who were very happy to be absolved of blame for the theory that it was the toxicity of established editors that was behind the deceleration and reversal of Wikipedia's editor numbers.

But there is good news for all guessers.

This myth was busted by a study done all way back in 2015. A literal Ice Age ago in Wikipedia years. It's findings were extremely revealing....

"Perhaps surprisingly, approximately 30% of attacks come from registered users with over a 100 contributions." In other words, a third of all personal attacks come from regular Wikipedia editors who contribute several edits per month. Personal attacks seem to be baked into Wikipedia culture.

The researchers also found that an outsized percentage of attacks come from a very small number of "highly toxic" Wikipedia contributors. A whopping 9% of attacks in 2015 came from just 34 users who had made 20 or more personal attacks during the year. "Significant progress could be made by moderating a relatively small number of frequent attackers," the researchers note. This finding bolsters the idea that problems in online communities often come from a small minority of highly vocal users.

.....

The algorithm was also able to identify a phenomenon often called the "pile-on." They found that attacking comments are 22 times more likely to occur close to another attacking comment. "Personal attacks cluster together in time," the researchers write. "Perhaps because one personal attack triggers another."

....

It means that an algorithm might be able to identify a pile-on before it really blows up, and moderators could come in to de-escalate before things get really ugly.

Depressingly, the study also found that very few personal attacks are moderated. Only 17.9% of personal attacks lead to a warning or ban. Attackers are more likely to be moderated if they have launched a number of attacks or have been moderated before. But still, this is an abysmal rate of moderation for the most obvious and blatant form of abuse that can happen in a community.

For probably quite understandable reasons, this study is not very well know on Wikipedia. It is of course well known to academics and indeed the Foundation, who hoped to use it as a means to create an automated moderation tool. And that is perhaps the only context I have seen it discussed on Wikipedia or its sister site Wikipediocracy, to lampoon those who would try try use technology to improve the Wikipedia environment. What fools they are!

Anyway, as we all know, those studies are totally out of date now. Civility is taken more seriously. Or seriously. OK they know of it now, and are capable of using the term in a non-derogatory way. Those results would not be replicated today.

Or would they?

*consults short term memory*

1, 2, 3, 4,.......Cullen328, Drmies, 32, 33, 34.

Oh dear.

That said, the confrontational nature of Wikipedia (which is inevitable by its design) is invariably going to put a lot of people off, regardless of whether the comments they get are toxic or not.

....but you do get that this specific study was about measuring the effects of toxic comments? And that it has seemingly proven "regardless of" is not remotely the right form of words here. Replace it with "but it does seem to matter" and you might convince people that you arguing with experts on Wikipedia pages is a good thing for humanity.

And fun fact. A revert button is like a pointy stick. It can be used for extreme violence or litter picking. Some fool back in the pre-history of Wikipedia seems to have wrongly assumed humans who willingly volunteered to create and maintain a public park, with the help of a free handy "How To Use Your Pointy Stick" leaflet and a friendly janitor or two wandering the park, could be trusted to handle a pointy stick. For the good of humanity.

On behalf of all humans, you have been such a disappointment, Hemiauchenia.

Now go get your stick, and stick it in your eye. Your Japseye.

Not even born with the brains of monkeys these people.

Now where did I put my Thesaurus.....

[ericbarbour]

Wikipedia does allow paid editing. And even though this is only permitted with full public disclosure, neither the Foundation or the volunteers have made any effort at all to determine what percentage of editors are paid and what percentage of Wikipedia content has been paid for. There is no central register of legitimate paid editors nor a database of articles featuring paid content. There is not even a standardised method of disclosure. And like everything on Wikipedia, hence the disclaimer, it is down to volunteers to do even the basics and alert readers to the fact they are reading an article that features paid content. The fact that Wikipedia allows paid editing is not even mentioned in the Wikipedia disclaimer.

Correct. And NO ONE wants to open THAT can of worms. It would result in a massive burst of bad publicity.

Even though Wikipedia content is possibly (see above) mostly viewed as the product of volunteers doing it out of only the goodness of their hearts, it is an undisputed fact that contrary to the founding principles of the so-called Wikipedia movement, it is now virtually impossible for them to create, distribute and facilitate the re-use of this content without the assistance and cooperation of the paid employees of the Wikimedia Foundation or use of its assets. In extremis, it would take millions of dollars and several years to successfully "fork" Wikipedia, even though the content is legally free to copy. Any study of Wikipedia's "community" of " volunteers" that neglects to account for these powerful levers of influence that rest in the hands of paid actors, is inherently flawed.

Many people have already grabbed copies and reposted them--almost always in read-only. Unless some billionaire starts a company to copy WP content and improve it, by actually PAYING someone to re-edit, nothing will happen. It's just too much work--for anything other than a large "nerd cult".