Posted: May 16, 2012 8:57 pm
I'm not 100% convinced about this paper, there seems to be a glaring methodological flaw in experiment 1, though I can't find a table of their data to see how much of a difference it makes.
The participants in the experiment were presented with cropped images from facebook users who identified themselves as gay or straight (which incidentally presents another interpretation than the one John P M identified, that people can identify "gay poses" rather than "gay faces"). The particpants then had to categorise each face as "straight" or "gay", meaning that there were 4 possible response-answer combinations:-
- Guess gay and "hit" correctly that the person identified as gay, H
- Guess gay and have a "false alarm" when the person identified as straight, FA
- Guess straight and "hit" correctly that the person identified as straight, H*
- Guess straight and have a "false alarm" when the person identified as gay, FA*
Now, there is a difficulty in treating this data to create a probability by chance of some given combination of guesses, so an aggregate measure A' is used which accounts for the fact that people may not estimate the correct proportion of the sample as being gay (it was in fact 50/50). A' is described here: http://www.amsciepub.com/doi/abs/10.246 ... 76.42.1.98
It is important to note that the test principle - that people can identify the difference between straight faces and gay faces - is neutral, if participants can tell them apart, they should do equally well both when they choose "straight" and when they choose "gay".
However, we notice that the only results presented relate to the smaller (and thus more susceptible to noise) data set of gay guess, H and FA. The other data is simply disregarded. This seems to be a grave concern, if the results had been different, there would have been nothing preventing the authors disregarding the data they presented and using the data they disregarded instead. This represents a selection bias and potential source of error, though it's impossible to tell without more information how serious this may be.
The participants in the experiment were presented with cropped images from facebook users who identified themselves as gay or straight (which incidentally presents another interpretation than the one John P M identified, that people can identify "gay poses" rather than "gay faces"). The particpants then had to categorise each face as "straight" or "gay", meaning that there were 4 possible response-answer combinations:-
- Guess gay and "hit" correctly that the person identified as gay, H
- Guess gay and have a "false alarm" when the person identified as straight, FA
- Guess straight and "hit" correctly that the person identified as straight, H*
- Guess straight and have a "false alarm" when the person identified as gay, FA*
Now, there is a difficulty in treating this data to create a probability by chance of some given combination of guesses, so an aggregate measure A' is used which accounts for the fact that people may not estimate the correct proportion of the sample as being gay (it was in fact 50/50). A' is described here: http://www.amsciepub.com/doi/abs/10.246 ... 76.42.1.98
It is important to note that the test principle - that people can identify the difference between straight faces and gay faces - is neutral, if participants can tell them apart, they should do equally well both when they choose "straight" and when they choose "gay".
However, we notice that the only results presented relate to the smaller (and thus more susceptible to noise) data set of gay guess, H and FA. The other data is simply disregarded. This seems to be a grave concern, if the results had been different, there would have been nothing preventing the authors disregarding the data they presented and using the data they disregarded instead. This represents a selection bias and potential source of error, though it's impossible to tell without more information how serious this may be.