Current Events > The Gender studies hoax - fake papers passing peer review

(+) Yeah! (+)

https://www.reddit.com/r/HarpiesBizarre/comments/9o1szg/gender_studies_hoax_three_scholars_wrote_20_fake/

Excerpt:

Other means superior to the natural sciences exist to extract alternative knowledges about stars and enriching astronomy, including ethnography and other social science methodologies, careful examination of the intersection of extant astrologies from around the globe, incorporation of mythological narratives and modern feminist analysis of them, feminist interpretative dance (especially with regard to the movements of the stars and their astrological significance), and direct application of feminist and postcolonial discourses concerning alternative knowledges and cultural narratives.

Oh dear
---

it sounds honestly like people in those fields doing peer review probably don't read the articles.

This was a big story last week or so.

Really a demonstration that certain fields need a boot in the ass.
---

Old
---

feminist interpretative dance

This I gotta see
---

what the fuck is that sub

This has been picked apart by a lot of scientists already. Basically:

- The people in this study do their best to misrepresent how many of their papers actually did get accepted, and how reputable the journals were
- They often invented fake data and tacked on weird ideas in the conclusion. When reviewers thought the data were interesting, they made it look as if those comments were directed at the weird ideas. Often the reviewers actually discarded the weird ideas as weird.
- Perhaps most notably, this study is in itself pretty bad science, as there is no comparison to any other field. Predatory journals exist in literally every field, and their method does not even consider that.
---

COVxy posted...

Really a demonstration that certain fields need a boot in the ass.

---

Broseph_Stalin posted...

what the fuck is that sub

A very toxic one, but i found this article interesting at least
---

Low tier journals will publish basically anything as long as they are paid for. As dumb as this is, it has very little bearing on the integrity of the field.

scar the 1 posted...

- Perhaps most notably, this study is in itself pretty bad science, as there is no comparison to any other field. Predatory journals exist in literally every field, and their method does not even consider that

This I don't buy at all. Several of the studies that did get accepted got accepted into reputable journals in the field.
---

COVxy posted...

scar the 1 posted...

- Perhaps most notably, this study is in itself pretty bad science, as there is no comparison to any other field. Predatory journals exist in literally every field, and their method does not even consider that

This I don't buy at all. Several of the studies that did get accepted got accepted into reputable journals in the field.

What aren't you buying? They didn't check vs a control group. That's very basic, and without it there's no conclusion to be drawn about the gender studies field. You know this. Fucking Nature published a paper on how water has memory.
---

scar the 1 posted...

What aren't you buying? They didn't check vs a control group. That's very basic, and without it there's no conclusion to be drawn about the gender studies field. You know this. Fucking Nature published a paper on how water has memory.

And to the extent other fields have this same issue, there's trouble in those fields as well.
---

It sounds like the bar for accepted submissions needs to be raised if trolly fake subs are getting green lights.
---

COVxy posted...

scar the 1 posted...

What aren't you buying? They didn't check vs a control group. That's very basic, and without it there's no conclusion to be drawn about the gender studies field. You know this. Fucking Nature published a paper on how water has memory.

And to the extent other fields have this same issue, there's trouble in those fields as well.

Yeah, I agree. But they (and you) have no basis for singling out gender studies like this. In their case, they're very obviously being dishonest, implying an agenda to make the field look bad, and make it look like "less serious science".
In your case idk.
---

scar the 1 posted...

COVxy posted...

scar the 1 posted...

- Perhaps most notably, this study is in itself pretty bad science, as there is no comparison to any other field. Predatory journals exist in literally every field, and their method does not even consider that

This I don't buy at all. Several of the studies that did get accepted got accepted into reputable journals in the field.

What aren't you buying? They didn't check vs a control group. That's very basic, and without it there's no conclusion to be drawn about the gender studies field. You know this. Fucking Nature published a paper on how water has memory.

That's tame. Stanfords technology journal published a request to be removed from a mailing list and Cambridge university published a call to ban cards against humanity in a psych journal.

scar the 1 posted...

Yeah, I agree. But they (and you) have no basis for singling out gender studies like this. In their case, they're very obviously being dishonest, implying an agenda to make the field look bad, and make it look like "less serious science".
In your case idk.

I'm not saying I have any knowledge of the base rates in any other field, just that the observed base rate here is waaay too high.

If discrimination of hoax work and real work is this low, that spells trouble for any field.
---

knutjob posted...

scar the 1 posted...

COVxy posted...

scar the 1 posted...

- Perhaps most notably, this study is in itself pretty bad science, as there is no comparison to any other field. Predatory journals exist in literally every field, and their method does not even consider that

This I don't buy at all. Several of the studies that did get accepted got accepted into reputable journals in the field.

What aren't you buying? They didn't check vs a control group. That's very basic, and without it there's no conclusion to be drawn about the gender studies field. You know this. Fucking Nature published a paper on how water has memory.

That's tame. Stanfords technology journal published a request to be removed from a mailing list and Cambridge university published a call to ban cards against humanity in a psych journal.

da fuck
---

COVxy posted...

scar the 1 posted...

Yeah, I agree. But they (and you) have no basis for singling out gender studies like this. In their case, they're very obviously being dishonest, implying an agenda to make the field look bad, and make it look like "less serious science".
In your case idk.

I'm not saying I have any knowledge of the base rates in any other field, just that the observed base rate here is waaay too high.

If discrimination of hoax work and real work is this low, that spells trouble for any field.

It says more about the individual journals and very little about the field.

COVxy posted...

scar the 1 posted...

Yeah, I agree. But they (and you) have no basis for singling out gender studies like this. In their case, they're very obviously being dishonest, implying an agenda to make the field look bad, and make it look like "less serious science".
In your case idk.

I'm not saying I have any knowledge of the base rates in any other field, just that the observed base rate here is waaay too high.

If discrimination of hoax work and real work is this low, that spells trouble for any field.

Yeah, but what do you expect? Reviewers work for free, and rarely if ever have resources to re-run experiments and as such validate that data isn't faked. Scientific publishing is - and has always been - heavily reliant on trust. The "study" ITT abuses that fact to discredit particular fields of study.
---

knutjob posted...

It says more about the individual journals and very little about the field.

6/7 acceptances came from reputable journals in the field, iirc. Journals that sent out the articles to peer review with other real scientists in the field.

This is very different than just getting a paper accepted in a predatory journal.
---

I've seen people contest the reputation of those journals, but I'm not in those fields so idk.
---

COVxy posted...

knutjob posted...

It says more about the individual journals and very little about the field.

6/7 acceptances came from reputable journals in the field, iirc. Journals that sent out the articles to peer review with other real scientists in the field.

This is very different than just getting a paper accepted in a predatory journal.

My apologies. I assumed we were talking across the wider social studies field. I checked out the journals that published the hoaxes and they are both highly specific and absurd.

That said 7 acceptances in 25 years doesn't seem a high success rate. Unless I'm missing something else since its difficult to navigate reddit on my shitty old phone.

scar the 1 posted...

Yeah, but what do you expect? Reviewers work for free, and rarely if ever have resources to re-run experiments and as such validate that data isn't faked. Scientific publishing is - and has always been - heavily reliant on trust. The "study" ITT abuses that fact to discredit particular fields of study.

Peer review, in general, is fucked. There's competing incentives, lack of time spent providing an actual thorough read through the work, this reliance on "trust" which can roughly be translated to the "benefit of the prestige", etc...

There's very little chance that peer review does a good job arbitrating good science. The system needs to be revamped in some way. At the very least, making peer reviews public record would go a long way.

This is all to say, this is almost unrelated to the current topic. An article goes through a desk editor and some number of reviewers. Discrimination of complete bullshit from real work shouldn't be this tough, even in our current system. Good science vs bad science, well that's a tough judgement, requires very careful reading and pretty pure incentives. "Real" vs "pulled out the ass in a deliberately hoax-y way" is a different story.
---

Some of the sections in their papers were literally just word-swapped chapters of Mein Kampf.
---

scar the 1 posted...

This has been picked apart by a lot of scientists already. Basically:

- The people in this study do their best to misrepresent how many of their papers actually did get accepted, and how reputable the journals were
- They often invented fake data and tacked on weird ideas in the conclusion. When reviewers thought the data were interesting, they made it look as if those comments were directed at the weird ideas. Often the reviewers actually discarded the weird ideas as weird.
- Perhaps most notably, this study is in itself pretty bad science, as there is no comparison to any other field. Predatory journals exist in literally every field, and their method does not even consider that.

All this plus....

1. Calling this a "scientific journal" is a huge stretch. The fields in question are highly subjective and ideological and extremely difficult to falsify.
2. Peer reviews are not to reject articles just because they disagree with the contents.
---

OpenlyGator posted...

It sounds like the bar for accepted submissions needs to be raised if trolly fake subs are getting green lights.

They did get caught so the system works. Maybe it needs to be more watchful to catch them quicker.
---

Yeah, and I encourage you to look at the review comments of the accepted papers. I remember specifically the one about dog rape culture. The reviewers found the data that they had to be interesting. It was data about how dog owners reacted differently towards male-on-female dog humping vs male-on-male dog humping. Faked data, but still. Then they objected to the discussion where the really weird conclusions were. But you know the review process. It's a process intended to improve the paper, which is what the review comments reflect.

I agree that it would be good to open up the review process - some publishers are doing this - but ultimately the discrimination between real and hoax should happen elsewhere in the pipeline, imo.
---

scar the 1 posted...

I agree that it would be good to open up the review process - some publishers are doing this - but ultimately the discrimination between real and hoax should happen elsewhere in the pipeline, imo.

Yeah, I mean, this is typically the editor's job. But in this case, you have someone either not doing their job and just sending things straight out for peer review. Or someone who thinks that these ideas were plausible enough to be real. Either way, there's a problem.
---

COVxy posted...

scar the 1 posted...

I agree that it would be good to open up the review process - some publishers are doing this - but ultimately the discrimination between real and hoax should happen elsewhere in the pipeline, imo.

Yeah, I mean, this is typically the editor's job. But in this case, you have someone either not doing their job and just sending things straight out for peer review. Or someone who thinks that these ideas were plausible enough to be real. Either way, there's a problem.

Yeah, but this shows another flaw of this study. How do you define what constitutes an obvious hoax or not? When a paper seems to have an experiment with actually interesting results, only that the conclusion is way out there, how do you judge that? Because that's the case here, they mix plausible data with outlandish reasoning.
---

scar the 1 posted...

When a paper seems to have an experiment with actually interesting results, only that the conclusion is way out there, how do you judge that? Because that's the case here, they mix plausible data with outlandish reasoning.

I would reject it in the position of an editor. If the conclusions don't match the data, that's a severe flaw.

If I ran a pure behavioral study on visual detection, and then concluded about thalamocortical basis of attentional tuning, it would get desk rejected. Almost certainly. You can't just conclude anything if you have "data". Scientific reasoning is making reasonable conclusions in the light of data. That's all science is.
---

COVxy posted...

scar the 1 posted...

When a paper seems to have an experiment with actually interesting results, only that the conclusion is way out there, how do you judge that? Because that's the case here, they mix plausible data with outlandish reasoning.

I would reject it in the position of an editor. If the conclusions don't match the data, that's a severe flaw.

If I ran a pure behavioral study on visual detection, and then concluded about thalamocortical basis of attentional tuning, it would get desk rejected. Almost certainly. You can't just conclude anything if you have "data". Scientific reasoning is making reasonable conclusions in the light of data. That's all science is.

Yeah so where do you draw the line? If some of the conclusions are wild and some aren't? Etc. Again, the goal of these papers was always to make them seem as publishable as possible, not to make them look clearly and obviously fake. Most of them were rejected, and several of them got severe criticism in the review stage.

But yes, of course an editor should clearly reject someone who has a conclusion that is completely disjointed from the results.
---

TBH, the big question this raises for me isn't about the individual journals, or the field.

It's about the internal review process. If it's this easy to get published, why do professors drag their feet for so long to put out publications on existing work? A lot of grad students' careers stagnate as their profs act like the review process is the equivalent of walking through Dante's Inferno, when in reality, as stated, there is heavy reliance on trust.
---

Over the past 12 months, three scholarsJames Lindsay, Helen Pluckrose, and Peter Boghossianwrote 20 fake papers using fashionable jargon to argue for ridiculous conclusions, and tried to get them placed in high-profile journals in fields including gender studies, queer studies, and fat studies.

There's a fat studies now?
---

scar the 1 posted...

Yeah so where do you draw the line? If some of the conclusions are wild and some aren't? Etc. Again, the goal of these papers was always to make them seem as publishable as possible, not to make them look clearly and obviously fake. Most of them were rejected, and several of them got severe criticism in the review stage.

Okay, if we take this logic forward a bit, let's assume all journals that rejected were good journals, let's excluded the journal that is a known predatory journal, and let's exclude the paper that used fake data. So then we are left with 5/18 acceptances. That's a ~28% success rate. So either that's the baseline for false positives, or 28% of the time these uneducated lay people submitted real honest to god good work accidentally while trying to submit a hoax. The latter seems unreasonable.

But really, to take this argument further, I would need to do more research and actually read the hoax submissions, prior and post peer review, to see where things might have gone wrong.

Darkman124 posted...

TBH, the big question this raises for me isn't about the individual journals, or the field.

It's about the internal review process. If it's this easy to get published, why do professors drag their feet for so long to put out publications on existing work? A lot of grad students' careers stagnate as their profs act like the review process is the equivalent of walking through Dante's Inferno, when in reality, as stated, there is heavy reliance on trust.

But that's just the thing. Publishing isn't a cake walk. It's a lot of time and effort with extremely hefty demands made by reviewers at each stage, at least in my field. My timecourse of publication attempts doesn't look too pretty:
https://imgur.com/aCDrebO

There's almost certainly field by field variation in the nature and rigor of peer review.
---

Yeah, I understand that your experience has been that 'this is not representative at all of how publishing is normally'

That has also been my experience. I think it is always dangerous to try to extrapolate that personal experience to define a broader trend, and it may not be. A broader study would be very useful to the industry of scientific publishing as a whole.

TBH, a study on resubmit rates for accepted publications would probably cover it for this.
---

COVxy posted...

scar the 1 posted...

When a paper seems to have an experiment with actually interesting results, only that the conclusion is way out there, how do you judge that? Because that's the case here, they mix plausible data with outlandish reasoning.

I would reject it in the position of an editor. If the conclusions don't match the data, that's a severe flaw.

If I ran a pure behavioral study on visual detection, and then concluded about thalamocortical basis of attentional tuning, it would get desk rejected. Almost certainly. You can't just conclude anything if you have "data". Scientific reasoning is making reasonable conclusions in the light of data. That's all science is.

What about if you submitted a paper with a study showing vaccinations cause autism?

COVxy posted...

So either that's the baseline for false positives, or 28% of the time these uneducated lay people submitted real honest to god good work accidentally while trying to submit a hoax. The latter seems unreasonable.

I don't agree with that binary dichotomy. But I agree you should read into it more if you're interested. From my perspective this was done by people with an agenda - to make "grievance studies" look as bad as possible. So they're not bound by trying to make a fair or repeatable study. Is there a problem within publishing? Yeah, we all agree on that. Is this problem bigger in gender studies? Nothing about this "study" actually supports that.
---

Darkman124 posted...

Yeah, I understand that your experience has been that 'this is not representative at all of how publishing is normally'

That has also been my experience. I think it is always dangerous to try to extrapolate that personal experience to define a broader trend, and it may not be. A broader study would be very useful to the industry of scientific publishing as a whole.

TBH, a study on resubmit rates for accepted publications would probably cover it for this.

I used to work as an editor for nature. Most papers went through 3 or 4 resubmits before they are accepted.

knutjob posted...

I used to work as an editor for nature. Most papers went through 3 or 4 resubmits before they are accepted.

I don't believe a single thing you post, ever.
---

Darkman124 posted...

knutjob posted...

I used to work as an editor for nature. Most papers went through 3 or 4 resubmits before they are accepted.

I don't believe a single thing you post, ever.

Huh? I barely post here

Targeting gender studies here shows an intense ignorance and a desire to highlight one field as sub par in this manner.

What do these fields have in common?

Chemistry? Physics? Biology? Gender studies?

If you guessed they all had fraudulent papers published in major journals that passed peer review, highlighting the inherent problems with scientific research and rigor in terms of analyzing new discoveries...

Congrats, you win.

Anyhow, no shit... some people published some fraudulent research... ok, welcome to the debate started back in the early 1990s, its be great if you caught yourself up a little further than cherry picking fields you dont like. Right now? This shit is way behind the curve. Mining peer reviewed journals that have debunked or nonsensical published data? Is fun... mining it and looking for one specific field and only that field?

Thats disingenuous, and kind of lazy, as literally every single scientific field you can think of, has had this problem.

From cold fusion, room temperature superconductivity, to gender studies.
---

PreacherBeelze posted...


The article linked is a little preachy but you'll note that it brings up the 1990s physics hoax as an example in the introduction.
---

Tag
---

https://areomagazine.com/2018/10/02/academic-grievance-studies-and-the-corruption-of-scholarship/

It is a VERY long read, but this is the complete write up about the process and should be read by anyone interested in the mechanisms of the study itself.
---

There was another paper for a feminist magazine that was literally an excerpt from Mein Kampf

You guys are just mad that they got caught.
---

Doesn't matter what department this happens in...definitely hurts the whole peer review industry...not that it wasn't already hurting.

I think I remember reading one of the papers was discussing potentially having privileged students talked over in class...and maybe chained to the seat (it was something strange like this) and the reviewer said they didn't think they went far enough in their recommendations....

I'll have to read up on this thing again. It's probably pretty rampant in all disciplines, but those based less on hard science are probably easier to sneak them in.