fMRI: the new phrenology?

Studies of mental functions, behaviors and the nervous system.

Moderators: kiore, The_Metatron, Blip

fMRI: the new phrenology?

#1  Postby endless psych » Mar 13, 2010 8:01 am

Ok slightly sensationalist thread title: from Mind Hacks

A new study has looked at the reliability of fMRI brain scanning results over time, finding that the same experiment will only only be moderately reproducible when conducted at two different times, suggesting that fMRI is much less reliable than most researchers assume.
The authors of the paper are the same ones who brought us the study showing that it's possible to find 'brain activity' in a dead fish if the analysis is done in a way that is common but prone to false positives.
The paper will shortly appear in the Annals of the New York Academy of Sciences but they've put a copy online and, although it's a scientific article, it's remarkably easy to read.
They review all the studies to date on what is known as the 'test-retest reliability' of fMRI. This refers to the ability of a measure to give reproducible results.
For example, if you're measuring an adult's height you want to make sure that your tape measure gives you similar results each time you use it on the same person. Of course, you may have readings that vary by a millimetre or two each time, but if you get wildly different results on Monday and Tuesday, you probably want to bin your tape measure.
In fMRI there are two types of results. One is 'where in the brain' and the other is 'how strong' is the activity.
We can examine the first by looking at how well the active brain areas overlap in scans taken at two different times, and we can examine the second by looking at the similarity of the strength of the results using a statistical test like a correlation.
The better the overlap and the statistical relationship between the results from the same test on the same people at different times, the more we can rely on our measurement technique.
This new analysis reviewed all the previous studies that have looked at the test-retest reliability of fMRI and found that overall, active brain areas overlap about 30% of the time and the correlation for the strength of the activity was about 0.5. To get some perspective a result of 1 would indicate perfectly reliable and reproducible results while a result of 0 would indicate no reliability at all.
An overlap of 30% and a correlation result of 0.5 shows fMRI has moderate reliability, but is much poorer than most people assume.
However, this overall result is perhaps a little too broad, and the authors make the point that the reliability varies depending on the type of scanner being used, what test is being carried out by the participants, what brain areas are being investigated and how the results are analysed.
Indeed, a recent study on the test-retest reliability of fMRI studies of the 'reward system' found the reproducibility of the results to be worse than this general figure while another study found an auditory detection task produced better results.
The authors conclude:
One thing is abundantly clear: fMRI is an effective research tool that has opened broad new horizons of investigation to scientists around the world. However, the results from fMRI research may be somewhat less reliable than many researchers implicitly believe. While it may be frustrating to know that fMRI results are not perfectly replicable, it is beneficial to take a longer-term view regarding the scientific impact of these studies. In neuroimaging, as in other scientific fields, errors will be made and some results will not replicate. Still, over time some measure of truth will accrue. This chapter is not intended to be an accusation against fMRI as a method. Quite the contrary, it is meant to increase the understanding of how much each fMRI result can contribute to scientific knowledge.

Link to full text of paper (via @hysell).

So what do we reckon? Depending on what's being measured it seems that fMRI isn't all that great on the old inter-rater reliabilty...
Not down with the rock not down with the roll. SHUT UP GRAMMATIC OIL!
BLOG: ...and your electron microscope!
Website:The Twenty-first floor
User avatar
endless psych
Posts: 505
Age: 40

Scotland (ss)
Print view this post

Re: fMRI: the new phrenology?

#2  Postby Lazar » Mar 13, 2010 4:24 pm

natselrox gave a link to attempts to increase the reliability somewhere. I will see if I can find it and get back to you.

EDIT: Here it is folks

natselrox wrote:This might interest you, Lazar. The point we were both trying to make.

How reliable are the fMRI results?

Functional magnetic resonance imaging is one of the most important methods for in vivo investigation of cognitive processes in the human brain. Within the last two decades an explosion of research has emerged using fMRI, revealing the underpinnings of everything from motor and sensory processes to the foundations of social cognition. While these results have revealed the potential of neuroimaging, important questions regarding the reliability of these results remain unanswered. In this chapter we take a close look at what is currently known about the reliability of fMRI findings. First, we examine the many factors that influence the quality of acquired fMRI data. We also conduct a review of the existing literature to determine if some measure of agreement has emerged regarding the reliability of fMRI. Finally, we provide commentary on ways to improve fMRI reliability and what questions remain unanswered. Reliability is the foundation on which scientific investigation is based. How reliable are the results from fMRI?

pdf here

Spinozasgalt: "And how come no one ever sigs me?"
User avatar
Posts: 2280
Age: 38

Australia (au)
Print view this post

Return to Psychology & Neuroscience

Who is online

Users viewing this topic: No registered users and 1 guest