Current Events > The Absurdity of Peer Review

(+) Yeah! (+)

https://elemental.medium.com/the-absurdity-of-peer-review-1d58e5d9e661
Some interesting excerpts:

Peer review has a misleadingly long history, conveniently dated to when the Royal Society started asking for anonymous reports in the 1830s on whether some papers should be published in its Philosophical Transactions. But the use of peer review was patchy at best until the 1970s, when it became the norm at all kinds of scientific journals. Indeed the journal Nature, for many the pinnacle venue for publishing science, didnt start peer review until 1973. In the short span of time since the 1970s, peer review has now grown from an occasional things into a beast that is devouring the time and energy and careers of scientists.

Pandemics dont conveniently hang around waiting for that slow, grinding peer review process to judge science. Science in the time of Covid-19 has had to be nimble, quick on its feet, has had to show its findings to the world without the layers of review-and-revise. Weve needed rapid research into models of transmission, into immunity and reinfection, into public health messaging and effective interventions, into drugs to treat symptoms, machines to support the severely ill, and the development of radically new types of vaccine. So pre-prints, those manuscripts put on the internet for all to read before peer review, became the weapon of choice. Long common in physics, pre-prints in biology and especially medicine exploded in number during the pandemic.

And the media have dealt with this explosion by consistently pointing out when research has not yet been peer reviewed. Presumably they do this to warn the reader that the research lacks the safeguards that peer review brings. The problem with that warning is peer review guards against nothing.

---

My chemistry teacher would give this article a paddling

---

What are you trying to peddle here? Standard science doubting?

---

So you want scientific journals to just publish whatever someone decides to write that day?

---

LordFarquad1312 posted...

So you want scientific journals to just publish whatever someone decides to write that day?

I think the peer review/journal system is on the way out, tbh. I think most science will continue to be self regulated by scientists the same way preprints work now-a-days. I almost exclusively read preprints now.

---

AldousIsDead posted...

What are you trying to peddle here? Standard science doubting?

...isn't that the entire point of peer review?

yes my dear... undermine the process.. only then will i be able to publish my treatise on message board ontology and high entropy coherency semantics

---

MedeaLysistrata posted...

only then will i be able to publish my treatise on message board ontology and high entropy coherency semantics

why not go straight to Timecube?

Questionmarktarius posted...

...isn't that the entire point of peer review?

Fair, but you know what I mean.

---

Peer review has kind of become a giant circlejerk where researchers reference the papers of the people doing peer reviews, who then approve those papers and then reference said paper in their next publication. Half the time the people doing the actual review barely understand the concept being published and are just giving it a sniff test to see if it reads as legit. The fact that we've had multiple examples now of complete nonsense being written (or literally generated) and making it past peer review is evidence enough that the system is fundamentally broken.

Review is absolutely necessary, but the current model doesn't get the job done anymore.

---

s0nicfan posted...

Peer review has kind of become a giant circlejerk where researchers reference the papers of the people doing peer reviews, who then approve those papers and then reference said paper in their next publication.

The academia circle-jerk is the other side of that, yes.

Questionmarktarius posted...

why not go straight to Timecube?

that's the big leagues, one thing at a time

---

COVxy posted...

Weve needed rapid research into models of transmission, into immunity and reinfection, into public health messaging and effective interventions, into drugs to treat symptoms, machines to support the severely ill, and the development of radically new types of vaccine. So pre-prints, those manuscripts put on the internet for all to read before peer review, became the weapon of choice. Long common in physics, pre-prints in biology and especially medicine exploded in number during the pandemic.

Interesting. This seems to be the expected practice in economics. I wonder if that contributes to it's study being relatively popular song laypeople

---

MedeaLysistrata posted...

that's the big leagues, one thing at a time

Fair

s0nicfan posted...

Peer review has kind of become a giant circlejerk where researchers reference the papers of the people doing peer reviews, who then approve those papers and then reference said paper in their next publication.

Pragmatically that doesn't make sense. But the general idea is not entirely false. Reviewers will often push authors to cite their own studies, only if peripherally related (though it's easy to just say no to this and the editor will almost never give a shit).

---

s0nicfan posted...

Review is absolutely necessary

I'm also not convinced that's true.

---

up

---

I wouldn't call it absurd, I would call it a necessary step in scientific research that is handled very badly

---

indica posted...

I wouldn't call it absurd, I would call it a necessary step in scientific research that is handled very badly

Is there a better way to handle it other than to not handle it at all?

---

COVxy posted...

Is there a better way to handle it other than to not handle it at all?

So you think scientific research would be better if it wasn't reviewed by other scientists?

---

Peer review is literally an integral pillar of science.

What the fuck is this topic even?

indica posted...

So you think scientific research would be better if it wasn't reviewed by other scientists?

I think the system would work a lot better with far less unnecessary work and less harmful forms of gatekeeping if the norm were simply "preprints". I think, most of the time, very little is gained through peer review, presumably because the majority of scientists want to do good science and want other scientists to pick up and use their work, so what gets written up has already been put through the wringer.

Good and interesting work will be picked up and "reviewed" anyway (don't get me wrong, still with harmful social biases, but no worse than what currently exists)

---

COVxy posted...

I think the system would work a lot better with far less unnecessary work

Again,

Saquon2k22 posted...

Peer review is literally an integral pillar of science.

What the fuck is this topic even?

Saquon2k22 posted...

Again,

I think the issue is that people think that peer review is something it isn't.

You get 2-3 highly overworked people who are at least peripherally familiar with similar work, and they spend as little time as possible reading your paper and jotting down questions and flaws that may or may not be valid. See, there's no expectation of quality with peer review, and since it is almost always anonymous, there's no cost to giving a shitty review. Then the editor reads over the reviews and tries to get a sense of how negative they are and makes the final decision.

Nobody gains anything from being a reviewer, and there's no quality control. So the output is 2-3 of the noisiest evaluations you could possibly get. Compound on top of this that there are substantial biases at play (reviewers know whose work their reviewing), and you have a recipe for disaster.

This whole process can take an extraordinary amount of time btw. Like the review process can extend to years. I mean, most of the time you get back first round in 1-3 months, and then second round depends on a lot of things, but can be about the same amount of time. But the whole process delays science getting into the literature by a substantial degree.

---

If you dont understand how science works, then just say that.

Reminds me of this clip:

https://www.youtube.com/watch?v=U5sRYsMjiAQ

"peer injunction"

To summarise the point if you don't want to watch, his work was severely stunted and slowed down because pretty much the only other person in the world who could understand what he was researching had a personal beef with him, and was more powerful politically.

People prop up the process a little too highly in the media. I guess there's not really a more effective way to do things, but the idea people have that it eliminates bias is wrong.

---

Does it stop a plainly wrong or plainly nonsense paper from being published? No. There will always be somewhere to publish a paper, so the mere act of publication is no guide to the quality of science. Not even in the most elite journals.

This is such a ridiculous argument against peer reviews. Just because journals with low standards exist, doesn't make the standard useless. If researchers have to turn to low quality journals to get their plainly wrong articles published, then peer review has clearly successfully prevented them from being published in the higher rated ones.

It's like saying locking your door doesn't work because burglars can just find another house with an unlocked door.

---

Saquon2k22 posted...

If you dont understand how science works, then just say that.

Lol. Whatever you want to believe.

---

Hornezz posted...

This is such a ridiculous argument against peer reviews. Just because journals with low standards exist, doesn't make the standard useless. If researchers have to turn to low quality journals to get their plainly wrong articles published, then peer review has clearly successfully prevented them from being published in the higher rated ones.

It's like saying locking your door doesn't work because burglars can just find another house with an unlocked door.

Crappy science gets published in high end journals all the time. Especially if it's written with flash and has the prestige to back it up.

---

COVxy posted...

Crappy science gets published in high end journals all the time. Especially if it's written with flash and has the prestige to back it up.

If you say so, random guy on the Internet.

COVxy posted...

Crappy science gets published in high end journals all the time. Especially if it's written with flash and has the prestige to back it up.

I feel like the article's author's argument is based on a giant straw man. People in the academic world understand that passing peer review doesn't automatically make a paper absolute truth, or free from errors and manipulation. Peer review just acts as a basic bullshit filter. The author does have a point that journalists in conventional media often don't understand this, but he should point his critique at them instead of the concept of peer review.

Also your claim of "all the time" is hyperbolic. Peer review does succeed in filtering out a lot of bullshit, even if some slips through the cracks. Removing the filter is only going to increase the amount of crappy science published.

---

Hornezz posted...

I feel like the article's author's argument is based on a giant straw man. People in the academic world understand that passing peer review doesn't automatically make a paper absolute truth, or free from errors and manipulation. Peer review just acts as a basic bullshit filter. The author does have a point that journalists in conventional media often don't understand this, but he should point his critique at them instead of the concept of peer review.

Also your claim of "all the time" is hyperbolic. Peer review does succeed in filtering out a lot of bullshit, even if some slips through the cracks. Removing the filter is only going to increase the amount of crappy science published.

By all the time, I mean about in equal proportion to what I see on preprint servers. The key point is that the prevalence of bullshit is very low. Sure, people could start posting pure bullshit all the time, but they won't. There's nothing to gain and a lot to lose.

Everyone whose gone through the process knows that it's unlikely to help, eats up an extreme amount of time and effort, and delays science substantially. And we all continue with the convenient fiction because this is, for many of us, how it has always been. But it doesn't have to be. And more and more scientists are primarily reading preprints now-a-days.

---

COVxy posted...

Sure, people could start posting pure bullshit all the time, but they won't. There's nothing to gain and a lot to lose.

Of course there's a lot to gain. Being published in a leading peer-reviewed journal gives a researcher status and can mean a significant boost in their career. It can help in getting research grants, or get them in better positions. Researchers use their publications for clout all the time. Once they realize they can get this based on pure bullshit, of course some would abuse the system.

Also, I've yet to read an alternative. Should the journals just publish anything that's sent to them with no selection process? Without a way to filter good papers from the bad ones, what would even be the point of quality journals in the first place? Researchers can already just publish their junk on their own blogs and be free from any barriers or criticism from their peers. I fail to see how less scrutiny would be beneficial to the scientific world.

Everyone whose gone through the process knows that it's unlikely to help,

Surely someone with a scientific background wouldn't just use "everyone knows" as an argument. Especially when it's blatantly false.

---

COVxy posted...

I think the peer review/journal system is on the way out

Hornezz posted...

Of course there's a lot to gain.

You misunderstood me. I said there is not much to gain from posting bullshit onto preprint servers. No one will read it, and to the extent to which people do it'll only hurt your reputation. See, the nice thing about preprint servers is that they don't have conferred prestige, so there's nothing you gain unless you post cool (read: good) science.

Hornezz posted...

Also, I've yet to read an alternative. Should the journals just publish anything that's sent to them with no selection process? Without a way to filter good papers from the bad ones, what would even be the point of quality journals in the first place? Researchers can already just publish their junk on their own blogs and be free from any barriers or criticism from their peers. I fail to see how less scrutiny would be beneficial to the scientific world.

An alternative, like I suggested, is to do away with the journaling system all together and just use centralized preprint servers (which already exist and are widely used, see arxiv and biorxiv). There's no strong evidence that peer review is beneficial (and the little evidence we have suggests the benefit is very small and primarily due to formatting/production). But the strain of peer review on the system is giant.

And it's not like these are controversial views! Scientists commonly state this much all the time. (See OP, see pinky's post, look on twitter)

Hornezz posted...

Surely someone with a scientific background wouldn't just use "everyone knows" as an argument. Especially when it's blatantly false.

It's exhausting having to justify my expertise to every person on this forum that disagrees with me. Go search the logs if you want. Either way, believe me or don't, it's not really worth my effort to prove it to you.

---

system worked fine until the (social)media starts cherry-picking brand new papers and banding them about is proven fact. was it perfect, no, nothing is. but the alternative is, well, the current mess we've got going on now, where bad science 'papers' are released and get snapped up because they wrote a catchy title and dumbasses want to use it to prove their point on twitter...

---

COVxy posted...

I think the peer review/journal system is on the way out, tbh. I think most science will continue to be self regulated by scientists the same way preprints work now-a-days. I almost exclusively read preprints now.

Scientists will throw out any "breaking" study for money. It's an issue in the psychology field where people just throw out articles to get that government funding for example.

---

COVxy posted...

You misunderstood me. I said there is not much to gain from posting bullshit onto preprint servers.
No one will read it, and to the extent to which people do it'll only hurt your reputation. See, the nice thing about preprint servers is that they don't have conferred prestige, so there's nothing you gain unless you post cool (read: good) science.

None of what you said here makes sense. "No one will read it"? that's exactly the point of peer review, to prevent bad science articles from being spread everywhere. If you remove the selection process for publication then readers have no choice but to read it themselves before being able to judge its quality. It's doing the exact opposite of what you claim.

Scientists gain reputation from being published in leading peer reviewed journals. If we do away with quality checks and every paper is treated equally, then there's nothing separating "cool science" from the uncool science - and also no reputations to be hurt.

It's exhausting having to justify my expertise to every person on this forum that disagrees with me. Go search the logs if you want. Either way, believe me or don't, it's not really worth my effort to prove it to you.

I went through the process of peer review myself (I've got a whopping 1 lol publications to my name), and I consider it helpful. As did all of the people I worked with. Just claiming "everyone knows" is not just a common belief fallacy, it's also not true.

look on twitter
Go search the logs if you want.

I you want to be taken seriously, you shouldn't post these sort of things.

---

This thread man damn.

Jesus, some of you guys can't even have a benign discussion about academic procedure without getting toxic as fuck

---

Questionmarktarius posted...

why not go straight to Timecube?

lol

---

COVxy posted...

An alternative, like I suggested, is to do away with the journaling system all together and just use centralized preprint servers (which already exist and are widely used, see arxiv and biorxiv). There's no strong evidence that peer review is beneficial (and the little evidence we have suggests the benefit is very small and primarily due to formatting/production). But the strain of peer review on the system is giant.

@TheLesserFaithX

---

I decided not to pursue wildlife biology because the amount of politics that go into publishing a paper is astonishing

Biologists have been suppressing information that doesn't fit their career goals for 200 years

Hornezz posted...

None of what you said here makes sense. "No one will read it"? that's exactly the point of peer review, to prevent bad science articles from being spread everywhere. If you remove the selection process for publication then readers have no choice but to read it themselves before being able to judge its quality.

I encourage you to take a look at preprint servers. There's a reason that there's an extremely skewed set of read and engagement statistics across different papers. (This also predicts later citations after publication!)

I think you have just not a lot of experience in this domain.

Hornezz posted...

If we do away with quality checks and every paper is treated equally, then there's nothing separating "cool science" from the uncool science

How about the quality of the science!?

It also removes terrible heuristics from dominating in the job market (such as counting the number of Nature/Science pubs, or calculating average IF), forcing hiring committees to actually read the papers of their candidates.

Hornezz posted...

I you want to be taken seriously, you shouldn't post these sort of things.

There's serious academic discussion goes on on Twitter. Here's an article:
https://t.co/UMNPZ3dffz

Like discussed there, Twitter find its place as a constantly evolving scientific conference which you curate.

---

Peer reviewed is very important. Its gives a basis behind the methods and hypothesis being tested. Its what makes the research scientific. If you think its dumb and unimportant, science is not for you. I loved doing research.

---

Anyway peer review is good as a concept but it needs to be lean and there needs to be more emphasis on replication. How someone got their data should be clear and easy to reproduce. And if there's any code, it better be simple and easy to read for the love of God it's 2021


---

In my field, which is "government" basically, gray literature is the standard. A lot of quality research isn't peer reviewed; like stuff the BEA puts out to CBO studies to the CRS etc


---

Balrog0 posted...

In my field, which is "government" basically, gray literature is the standard. A lot of quality research isn't peer reviewed; like stuff the BEA puts out to CBO studies to the CRS etc

what are these acronyms and how is it be known to be quality research if it isn't passing a review process of some kind to make sure results can be replicated, etc?

---

Bureau of economic analysis
Congressional budget office
Congressional research services

The same way we know journals are good, reputation

---

pure_temper posted...

Anyway peer review is good as a concept but it needs to be lean and there needs to be more emphasis on replication. How someone got their data should be clear and easy to reproduce. And if there's any code, it better be simple and easy to read for the love of God it's 2021

You'd have better luck convincing academics to quit and become farmers.

---