Posted: Oct 06, 2013 3:47 pm
Trying to keep things in everyday language:
To me, for some effect to be realistically claimed to occur at better-than-chance levels, it should be possible for someone claiming that that happens to describe a situation where they are highly confident that better-than-chance behaviour *will* be observed by future unbiased experimenters if they do N trials each of experiment E with S subjects, whether subjects were chosen at random of provided as supposed talented people.
Now, it could be that the situation described would be impractical (requiring thousands of subjects spending days each in experiments), but if there actually was an effect, some situation should be at least describable, and the easiest situation someone was prepared to bet on would give a good illustration of the supposed size of the effect.
If someone making the claim for the effect existing can't or won't describe a situation they are prepared to make a serious bet on, to me that would suggest a lack of confidence in the effect existing despite the claims.
Similarly, if someone had done a meta-analysis and ended up claiming positive results, the experiment they are prepared to bet on clearly should relate fairly closely to the size of positive results they were claiming and the claimed confidence level in those results.
To me, for some effect to be realistically claimed to occur at better-than-chance levels, it should be possible for someone claiming that that happens to describe a situation where they are highly confident that better-than-chance behaviour *will* be observed by future unbiased experimenters if they do N trials each of experiment E with S subjects, whether subjects were chosen at random of provided as supposed talented people.
Now, it could be that the situation described would be impractical (requiring thousands of subjects spending days each in experiments), but if there actually was an effect, some situation should be at least describable, and the easiest situation someone was prepared to bet on would give a good illustration of the supposed size of the effect.
If someone making the claim for the effect existing can't or won't describe a situation they are prepared to make a serious bet on, to me that would suggest a lack of confidence in the effect existing despite the claims.
Similarly, if someone had done a meta-analysis and ended up claiming positive results, the experiment they are prepared to bet on clearly should relate fairly closely to the size of positive results they were claiming and the claimed confidence level in those results.