Sports Science using invalidated statistical methods

[Ronald Risher]

Cold water poured on scientific studies based on 'statistical cult'

Top statisticians are asking serious questions about dozens of research projects carried out by Australian sports scientists using a controversial method.

www.theage.com.au

I wonder how prevalent this is within the AFL system?

 

[Furn2]

The problem as a scientist, i'm now a former scientist is, you have to keep looking for funding and to do that you need to publish and you need interesting findings or you get your funding from someone with an agenda and you're pressured to use a method that fits that agenda. Eg finding that foam rollers or turmeric work.

In the AFL there isn't that incentive to produce or use dodgy findings as its counter productive , but there's still plenty of info out there that's taken as fact even by experts that's never actually been proven.

So yeh, they'd be doing plenty of things that arent actually effective based on poor research but they don't have the time or resources to work out which is which.

 

[lamby]

The article is a bit sensational.

The point of the methodology is that in sports sometimes a 0.5% gain can win you a gold medal. And an athlete could use something NOW that would give them edge. They can’t wait for a full double blind peer reviewed paper - their competition may already be using it. So the burden of proof is less.

The trouble is that ‘people’ (media, amateurs) take the results from this ‘sports methodology’ as ‘proven’. Where in reality it still needs ‘real’ scientific methodology to prove it. (And gives weight to the Science Denier crowd who say “look - science does not work”)

AFL clubs probably have the money and incentive to try some un-proven (but legal) training/treatments if there is a chance of getting an edge if it does no harm and has a small probability of actually working.

 

[Soberian Tiger]

A journo writes a sensationalist article. Who'd a thunk it?

 

[Kappa]

A journo writes a sensationalist article. Who'd a thunk it?

Sounds like you're pouring cold water on journalists?

 

[Soberian Tiger]

Sounds like you're pouring cold water on journalists?

Quite a few journos need to pour cold water on themselves.

Regularly.

It might reduce the inflammation of their sensitivities...

The VU Dean is right.

 

[theyellowsash]

I think you guys are underselling how dangerous this could be particularly to the young players who will do what they are told without asking. Sure, if all it involves is some hippy onion juice it's unlikely to cause much harm. But what about when pseudo psychologists subject a whole group to a week's worth of psychologically damaging rituals, or when the doctors start injecting players with either illegal substances or substances not tested for use on humans, sourced from God knows where?

It's fine enough for an individual athlete (eg swimmer) to choose for themselves a new experimental method. But if I was the parent of a kid walking into an AFL club with their sports scientists controlling everything they do without really having an option to do otherwise (or else you might as well quit the career), I'd want to be damn sure the methods they are using have been tested on more than 10 people with a better than 50% margin for error

 

[estibador]

I think you guys are underselling how dangerous this could be particularly to the young players who will do what they are told without asking. Sure, if all it involves is some hippy onion juice it's unlikely to cause much harm. But what about when pseudo psychologists subject a whole group to a week's worth of psychologically damaging rituals, or when the doctors start injecting players with either illegal substances or substances not tested for use on humans, sourced from God knows where?

If those were actually being proposed as studies it's unlikely they would gain ethics approval in the first place (especially the Essendon example obviously). So they don't have much relevance to the choice of statistical methods in actual published studies.

I don't know much about MBI, but I've heard many journals have always been reluctant to publish papers that use it. Nothing really new here.

 

[Soberian Tiger]

The article is a bit sensational.

The point of the methodology is that in sports sometimes a 0.5% gain can win you a gold medal. And an athlete could use something NOW that would give them edge. They can’t wait for a full double blind peer reviewed paper - their competition may already be using it. So the burden of proof is less.

The trouble is that ‘people’ (media, amateurs) take the results from this ‘sports methodology’ as ‘proven’. Where in reality it still needs ‘real’ scientific methodology to prove it. (And gives weight to the Science Denier crowd who say “look - science does not work”)

AFL clubs probably have the money and incentive to try some un-proven (but legal) training/treatments if there is a chance of getting an edge if it does no harm and has a small probability of actually working.

This comment would have been much better if the term, 'science denier', had not been dragged into it.

We should all deny some science, as some science will be right at any given time, while some science will be wrong. Equally, we should deny some journalism for the same reason; some at any given time will be right and some will be wrong. Both suffer from the fallibility of humanity. The difficulty is in deciding which is right and which is wrong.

That I decry the use of 'science denier' should in no way infer that I am of the political right, either. I am a former Laborite who chose the image of an old-time Laborite for my avatar. As a Laborite, I wrote free political content for the newspaper in question. That I am no longer a Laborite came about partly because I cannot be party to any political party that uses 'science denier' as an insult.

That, in no way, makes me anti-science, however. I fully expect all of my offspring to eventually hold a science degree of some sort. One already is a sports scientist of some distinction, who would surely agree that the use of the said methodology may be either right or wrong, depending upon the circumstances involved.

The internet age has surely spawned a fashion for the avoidance of the recognition of such grey areas. Almost everything seems to be reduced to black and white. The article did just that. Unfortunately, the use of 'science denier' did likewise.

 

[Duckimus Prime]

The VU Dean is right.

Is that the Dean that makes a statement about the work of Associate Professor Kristin Sainani only to have Associate Professor Kristin Sainani call the Dean's statement a "distortion of reality"?

 

[Tippett2Goal]

The article is a bit sensational.

The point of the methodology is that in sports sometimes a 0.5% gain can win you a gold medal. And an athlete could use something NOW that would give them edge. They can’t wait for a full double blind peer reviewed paper - their competition may already be using it. So the burden of proof is less.

he problem as a scientist, i'm now a former scientist is, you have to keep looking for funding and to do that you need to publish and you need interesting findings or you get your funding from someone with an agenda and you're pressured to use a method that fits that agenda.

As someone currently working in science (not sports science), another issue is that you can often get positive results which are not statistically significant at 95%. Even assuming large sample sizes and good controls, sometimes you get to the end of a study and you have pretty good correlations, but nothing statistically significant.

Now assuming justifiable data manipulation doesn't get you significance (ie removing outliers, publishing at 90% CI, data smoothing via logs, etc), you can either publish with positive, interesting, but non-significant results, or you can flush the work down the toilet...

Pretty logical what most would choose to do.

edit: And in this case, because they are looking for such small changes, they already recognise that the chances of finding significance are next to 0, so they want some sort of other stats approach to back up the publishing of non statistically significant results.

 

[mouncey2franklin]

The average joe on the street knows nothing about the replication crisis in science.

1,500 scientists lift the lid on reproducibility - Nature

Survey sheds light on the ‘crisis’ rocking research.

www.nature.com

They are never going to look into the methodology of studies for themselves.

All they will do is parrot the experts, or parrot their favourite talkback radio presenter.

Or, more likely, they will binge watch another three or four hours of netflix before nodding off to sleep.

 

[D-N-R]

Fair point by the author to suggest that studies with a mere handful of test subjects lack scientific credibility.

Did they even have a control group with the turmeric experiment?

And a single study doesn't prove anything by itself. It only gets proven if the results can be replicated by someone else.