Current Events > New proposal to shift statistical threshold from p < .05 to p < .005

(+) Yeah! (+)

http://www.nature.com/news/big-names-in-statistics-want-to-shake-up-much-maligned-p-value-1.22375

Full preprint:
https://osf.io/preprints/psyarxiv/mky9j

Shifting alpha to a new arbitrary value is literally pointless, especially in the way they recommend.

If you are a scientist, you need to understand statistics and data analysis, that's really all. And I don't think that this is largely the issue (I think most do understand the statistics deeply), but it's a good scapegoat for the culture of pushing positive results and the journaling system, which are much more difficult to fix.
---

Literally why would you care?
---

SageHarpuia posted...

Literally why would you care?

Cuz I science often.
---

SageHarpuia posted...

Literally why would you care?

p-value cutoffs affect significance of findings, which in turn affect chances of being published in reputable journals, which in turn affect career prospects. So shifting the cutoff would substantively affect millions of researchers around the world (including myself).

Fisher never intended for the p-value to be used as a hard cutoff for meaningful findings but as one aspect to be considering among a bunch of other factors. It's too bad that that's what it's become.
---

i feel like the problem is not that .05 is too high

it's that media tends to exaggerate the significance of findings with p-values that evoke lesser confidence
---

I guess it's less laughable than the time someone published, in a pretty highly reputable (but not well regarded) journal, a suggestion to switch from p-values to confidence intervals, even though they are literally mathematically equivalent.
---

This is fantastic news and I hope it happens.

Transcendentia posted...

This is fantastic news and I hope it happens.

You're, uh, really bad at this.
---

I am kinda rusty on statistics, but isn't a p value of 0.005 way too low to be practical? Like thats a very small part for outliers and in practice it would almost include the whole population?
---

I think 0.01 or 0.02 would be more reasonable. 0.005 is 10 times more significant. That's ridiculous with some distributions.
---

Transcendentia posted...

This is fantastic news and I hope it happens.

We've all had enough of experts.

you guys mad?

JM_14_GOW posted...

I am kinda rusty on statistics, but isn't a p value of 0.005 way too low to be practical? Like thats a very small part for outliers and in practice it would almost include the whole population?

iirc it would correspond to 3-sigma instead of 2-sigma on a normal probability distribution

it's not 'too low to be practical' so much as 'unnecessary and likely to suppress useful data'
---

Darkman124 posted...

JM_14_GOW posted...

I am kinda rusty on statistics, but isn't a p value of 0.005 way too low to be practical? Like thats a very small part for outliers and in practice it would almost include the whole population?

iirc it would correspond to 3-sigma instead of 2-sigma on a normal probability distribution

it's not 'too low to be practical' so much as 'unnecessary and likely to suppress useful data'

Yeah thats more or less what I am trying to say, basically its including the 99.995 of the population and you lose a significant amount of outliers imo.
---

JM_14_GOW posted...

Yeah thats more or less what I am trying to say, basically its including the 99.995 of the population and you lose a significant amount of outliers imo.

p<.005 is equivalent to a 99.5% confidence interval, which in a normal probabiltiy distribution is three standard deviations from the mean

it's common for engineering projects to require 3-sigma for safety of designs as it's treated as being as close to "we know this won't happen" as is possible, since the standard deviation there is typically a manufacturing tolerance.

i think it's a lot less viable in pure science where the error is influenced by things outside your control and cannot be as easily minimized
---

Nah.
---

Hmm. Tagging for interest later.
---

Darkman124 posted...

it's not 'too low to be practical' so much as 'unnecessary and likely to suppress useful data'

This is pretty much the primary issue. The false negative rate in a lot of fields is a much larger issue than the false positive issue.

We really shouldn't be paying too much attention to arbitrary cut offs, tbh.
---

Up.
---

SageHarpuia posted...

Literally why would you care?

His user name is literally COVxy.
---

ZMythos posted...

I think 0.01 or 0.02 would be more reasonable. 0.005 is 10 times more significant. That's ridiculous with some distributions.

The fact that people are saying things like this shows how arbitrary it is. It really depends more on the context. For some areas, like maybe marketing, a p value of 0.1 is fine. In something like astrophysics the p value needs to be far lower than 0.05
---

Transcendentia posted...

you guys mad?

Oh man, you are not good at this
---

literal_garbage posted...

Transcendentia posted...

you guys mad?

Oh man, you are not good at this

Must be meta-trolling from the great Clad.
---

I dont like it.
---

Back to the p-hacking board.
---

I don't have a whole lot of knowledge on this subject, but from my perspective, it seems like different fields would want to use different p value thresholds. It doesn't make sense to have a universal threshold that applies across all fields.
---

I thought the threshold varied depending on the field and type of study, and that 0.05 was more of an informal standard than anything?
---

Sativa_Rose posted...

I don't have a whole lot of knowledge on this subject, but from my perspective, it seems like different fields would want to use different p value thresholds. It doesn't make sense to have a universal threshold that applies across all fields.

Meh, it's more like a careful consideration of an argument in the context of the totality of the evidence is more important than an arbitrary cut off on any one of the tests.

I'd be completely fine accepting a paper with no statistically significant results if all of the data pointed towards a single conclusion, if there were enough convergent evidence.
---