Current Events > Peer review doesn't prevent bad science from being published.

(+) Yeah! (+)

Relevant twitter thread:
https://twitter.com/micahgallen/status/1462018084278083587?t=-_ZdSuA8_3jhKKf3JP_b-Q&s=19

I think one of the inconvenient truths we tell ourselves is that peer review keeps poorly conducted or "bad" studies out of the literature. I don't think that's true - in fact, I can't think of a single paper whose rejection I reviewed that didn't appear elsewhere.

I think we all know what happens to "good" and "bad" papers that are rejected: authors submit them elsewhere. Oftentimes, with little to no changes. Of course, I do believe that some papers are made better over the course of resubmission. Some are also made worse.

But ultimately, I'd wager that at the limit, a very small proportion of submitted and rejected papers don't end up published sometime, somewhere. So what does our simplistic reject/revise/accept system actually do?

One possibility, which seems likely to me, is that it primarily serves to reinforce existing power structures. That is - it enables scientists to push work they don't like to lower impact journals. That may or may not reinforce impact factors, but does it help science?

But if the basic premise of peer review - that it keeps bad science out of the literature - is false, I think it's cause to truly rethink our approach to scientific publishing.

[continued...]

Something to think about when you read a news article and they say "this science has/hasn't been peer reviewed".

---

Who is that one cemen going off incessantly about peer reviewed vaccine studies?

Being published elsewhere doesn't always get the same reach/credibility though.

But yeah a lot of work needs to be done on filtering out junk studies and politically tainted science.

---

So hes saying its bad because non-reviewed and rejected papers still get published? That doesnt negate why peer-reviewed scientific articles are important. In fact, you might say that just shows how important it is to look for something to BE reviewed before considering it as a source.

pikachupwnage posted...

Being published elsewhere doesn't always get the same reach/credibility though.

But yeah a lot of work needs to be done on filtering out junk studies and politically tainted science.

I think, for the most part, peer review does very little but gatekeeping prestige. Has very little to do with quality.

---

Up

---

COVxy posted...

I think, for the most part, peer review does very little but gatekeeping prestige. Has very little to do with quality.

I mean, it also makes sure they followed all the pertinent rules for studies and such. Which is why that hydroxychloroquine study that was cited by Trump has never received the tag of being peer reviewed. Because the study itself was flawed.

daynlokki posted...

I mean, it also makes sure they followed all the pertinent rules for studies and such.

What rules?

---

What_ posted...

Who is that one cemen going off incessantly about peer reviewed vaccine studies?

Joe40001
---

Let's not taint my topic with joenumbers lol

---

I'd rather see a flawed peer review system than no peer review system when it comes to scientific studies personally.

---

I do most of my reading of the scientific literature now-a-days from non peer reviewed preprints. From what I can tell, this is the case for most of my colleagues as well.

---

https://www.youtube.com/watch?v=N06fZxoUQx0

---

I imagine theres less fake bullshit in peer reviewed journals than there are in tabloids

Kloe_Rinz posted...

I imagine theres less fake bullshit in peer reviewed journals than there are in tabloids

Ah yes, the two natural modes of scientific literature publication.

---

Ysmir posted...

Ah yes, the two natural modes of scientific literature publication.

Well, what are you suggesting is the middle ground here

Yeah I feel like the premise of the original post is a little dumb.

"It's so easy to get published and be peer reviewed eventually, so keep that in mind about the stuff that apparently fail to even meet that threshold".

---

I think the "everything gets published eventually" is not the only part of the argument, only leading up to the real conclusion that peer review mostly serves the purpose of gate-keeping rather than assessment.

---

The point is that if youre reading from second or third tier journals, the fact that the papers are peer reviewed might say little-to-nothing about the quality of the research methods.

---

It prevents bad science from being published in publications with esteem and respect within their fields. Since online (and offline) journals that accept what the fuck ever exists, it doesn't keep bad science from being published. Nor has anyone who knows what they're talking about made the claim that it does. Your crusade against peer review continues to be the weirdest gimmick and I look forward to the endgame of your plan where you reveal yourself to be a natural healing guru who is mad that his theory of telepathy is rejected by the scientific community, so that this whole thing you got going will finally make sense.

---

DeadBankerDream posted...

It prevents bad science from being published in publications with esteem and respect within their fields. Since online (and offline) journals that accept what the fuck ever exists, it doesn't keep bad science from being published. Nor has anyone who knows what they're talking about made the claim that it does. Your crusade against peer review continues to be the weirdest gimmick and I look forward to the endgame of your plan where you reveal yourself to be a natural healing guru who is mad that his theory of telepathy is rejected by the scientific community, so that this whole thing you got going will finally make sense.

Did you know that what predicts whether an article will be published in the top tier journals is whether one of the authors has previously published in that journal?

Did you know that systematic observation of articles before and after peer review shows very little improvement?

---

I have never claimed peer review is perfect. Nothing I've seen you post suggests you want to see it improved. Rather, you'd prefer to burn it to the ground and replace it with nothing.

Not sure why peer review would improve articles. That seems to be up to the writer of said article to either take the review to heart or not.

---

Improvement of the system would require at least tripling the burden of an already overburdened system, which if even possible would substantially slow scientific progress down even more.

The way to improve biased and low reliability assessment is to increase the number and diversity of reviewers, and that's just not possible.

At the moment, peer review contributes far more harm to science than it does help. It not only perpetuates systematic biases through active gatekeeping, but produces substantial burden on scientists (who volunteer their time for free to peer review) and slows scientific progress by making results available to scientists on the order of years after the work is complete.

Improvement of work post peer review is absolutely expected given the structure of how the system works.

---

Stop treating journals like gospel, and open the gates to some sort of open-source wild-west sort of scientific shitpost pile.

Bullshit "studies" are already getting out into the wild and spread all over anyway, so go ahead and embrace it. Peer review will still happen, just in full public view. May even reduce general dumbassery, as the process of disproving garbage science is open for all to see.

This really is an impressive topic in many ways. We are seeing someone say complete gibberish with full conviction basically.

---

Whenever the topic of peer review comes up I think about this clip

https://www.youtube.com/watch?v=U5sRYsMjiAQ

Whatever you think of the Weinstein brothers I think they make a good point here. "peer injunction". I was discussing this with a cancer research scientist friend who agreed unilaterally with this.

---

Peer review doesn't prevent bad science from being published.

no shit

just like a hundred professional critics all immediately giving a new movie 5/5 stars does NOT mean its a legit good flick that the majority public is actually raving about...

often times it just means shilling hard for money is not a lost art


---

Yeah they may get published somewhere else but the big journals didn't publish them.

But the other side of gatekeeping like that is that it can lead to groupthink...

---

Peer review needs work, but I'd rather academia rally around stopping paywalling. The fact that legitimate studies will get paywalled and the authors don't see a dime and all they get in exchange is the privilege of saying they were in that journal is a problem.



---

We are seeing someone say complete gibberish with full conviction basically.

lol it's literally just you dude. don't be so threatened by people talking about stuff you don't get. probably try asking more and saying less.

---

s0nicfan posted...

Peer review needs work, but I'd rather academia rally around stopping paywalling. The fact that legitimate studies will get paywalled and the authors don't see a dime and all they get in exchange is the privilege of saying they were in that journal is a problem.

There are usually fees that the authors have to pay to the journal to get published. Unfortunately, the prevailing way in which the journal system allows open access is by shunting even more cost onto the authors.

Really, the journal system just needs to go.

---

I'm not saying just use sci-hub, but just use sci-hub.

---

COVxy posted...

I do most of my reading of the scientific literature now-a-days from non peer reviewed preprints. From what I can tell, this is the case for most of my colleagues as well.

I'm willing to bet that you know how to critically appraise most preprints, though. In a way, because of your credentials, you are subjecting it to a personal peer review.

---

I appraise preprints the same way I do published manuscripts.

Realistically, the scientific literature is very specific and technical, not sure anyone not trained in critically reading the literature really should be, anyway. That is to say, the scientific literature is for scientists.

---

This doesn't seem to be as much of a problem in math, mostly because the difference between "high quality" and "low quality" journals is the perceived importance of the results rather than anything to do with their validity.

---

Anteaterking posted...

This doesn't seem to be as much of a problem in math, mostly because the difference between "high quality" and "low quality" journals is the perceived importance of the results rather than anything to do with their validity.

Physics and math have been operating first and foremost out of arxiv for a long time, afaik. Right?

---

COVxy posted...

What rules?

If you dont know the answer for this already, then you arent equipped to make this topic

daynlokki posted...

If you dont know the answer for this already, then you arent equipped to make this topic

No need to lash out if you don't have an answer.

---

COVxy posted...

Physics and math have been operating first and foremost out of arxiv for a long time, afaik. Right?

For many things, but there's a slight air of "take everything on arXiv with a grain of salt" and for grants, etc. "real" publications are still weighted much higher.

Since I'm not in academia anymore, I'm still leaving some of my pre-prints solely on arXiv though.

---