by A new study has looked at the reliability of fMRI brain scanning results over time, finding that the same experiment will only only be moderately reproducible when conducted at two different times, suggesting that fMRI is much less reliable than most researchers assume.
The authors of the paper are the same ones who brought us the study showing that it's possible to find 'brain activity' in a dead fish if the analysis is done in a way that is common but prone to false positives.
The paper will shortly appear in the Annals of the New York Academy of Sciences but they've put a copy online and, although it's a scientific article, it's remarkably easy to read.
They review all the studies to date on what is known as the 'test-retest reliability' of fMRI. This refers to the ability of a measure to give reproducible results.
For example, if you're measuring an adult's height you want to make sure that your tape measure gives you similar results each time you use it on the same person. Of course, you may have readings that vary by a millimetre or two each time, but if you get wildly different results on Monday and Tuesday, you probably want to bin your tape measure.
In fMRI there are two types of results. One is 'where in the brain' and the other is 'how strong' is the activity.
We can examine the first by looking at how well the active brain areas overlap in scans taken at two different times, and we can examine the second by looking at the similarity of the strength of the results using a statistical test like a correlation.
The better the overlap and the statistical relationship between the results from the same test on the same people at different times, the more we can rely on our measurement technique.
This new analysis reviewed all the previous studies that have looked at the test-retest reliability of fMRI and found that overall, active brain areas overlap about 30% of the time and the correlation for the strength of the activity was about 0.5. To get some perspective a result of 1 would indicate perfectly reliable and reproducible results while a result of 0 would indicate no reliability at all.
An overlap of 30% and a correlation result of 0.5 shows fMRI has moderate reliability, but is much poorer than most people assume.
However, this overall result is perhaps a little too broad, and the authors make the point that the reliability varies depending on the type of scanner being used, what test is being carried out by the participants, what brain areas are being investigated and how the results are analysed.
Indeed, a recent study on the test-retest reliability of fMRI studies of the 'reward system' found the reproducibility of the results to be worse than this general figure while another study found an auditory detection task produced better results.
The authors conclude:
One thing is abundantly clear: fMRI is an effective research tool that has opened broad new horizons of investigation to scientists around the world. However, the results from fMRI research may be somewhat less reliable than many researchers implicitly believe. While it may be frustrating to know that fMRI results are not perfectly replicable, it is beneficial to take a longer-term view regarding the scientific impact of these studies. In neuroimaging, as in other scientific fields, errors will be made and some results will not replicate. Still, over time some measure of truth will accrue. This chapter is not intended to be an accusation against fMRI as a method. Quite the contrary, it is meant to increase the understanding of how much each fMRI result can contribute to scientific knowledge.
Link to full text of paper (via @hysell).
So what do we reckon? Depending on what's being measured it seems that fMRI isn't all that great on the old inter-rater reliabilty...
