Wednesday, February 16, 2011

Clinical Prediction Fails Again

I just gave the first Psych 144 exam of the semester. Before I graded them, though, I tried to predict everyone's score based on my experience talking with them, observing them work on activities, and so on. (I did not know how anyone had scored on quizzes or assignments because my TAs grade those.) My intuition told me that my predictions would be pretty good and I was expecting a correlation between my predictions and the actual scores of at least +.50 and probably higher.

The reality, though, was that the correlation was a measly +.17. (I won't show the scatterplot because people might be able to tell from it how I had predicted they would do. But the pattern in the data is barely detectable in it.)

To put things in perspective, I also computed the correlation between the order in which students finished the exam and their scores. It was -.40. (The negative correlation just means that students with lower ranks tended to score higher.) In other words, simply looking at the order in which students finished provided much better predictions than I did using my "professional judgment" based on experience.

[Consider also that this was on a day with horrendous traffic because of the "Get Motivated" event at the Save Mart Center, which I'm sure motivated a lot of complaints to the University Administration. Anyway, this made many students late and affected the order in which they finished. The correlation would probably have been stronger than -.40 otherwise.]

These results are actually typical of tons of research on "clinical prediction" (prediction done based on experience and intuition) and "actuarial prediction" (prediction based on a simple rule usually derived statistically from past data ... but in this case just pulled out of thin air). This research shows that although psychologists, doctors, teachers, and others tend to be quite confident in their clinical predictions, actuarial predictions are always more accurate overall.

One of the best books about psychology (in my opinion, of course) is House of Cards by Robyn Dawes, in which he discusses the problems with clinical prediction and their implications for clinical psychology. I strongly recommend it.

Dawes, by the way, is one of my own intellectual heroes. Unfortunately, he died just recently. (Here is an obituary for him.) Although I had chances to meet him, I never had the nerve to walk up and introduce myself. I definitely wish I had.

Friday, February 4, 2011

Alphabetical Order

Researchers Kurt Carlson and Jacqueline Conard at the Georgetown University School of Business published a very cool article in the Journal of Consumer Research. (By the way, much of the research that goes on in business schools is essentially applied psychology.)

They found that people whose last names are further toward the end of the alphabet respond more quickly to offers of free basketball tickets or an opportunity to complete a survey in return for a bottle of wine. Their theory is that people whose names are further toward the end of the alphabet have spent a lifetime being at the end of the line and waiting their turn, which causes them to jump more quickly at these kinds of opportunities.

Here is a more complete popular summary of this research and the research article itself.

There is really a lot to think about here. But one striking thing is the very respectable strength of the relationship. For example, the correlation between people's last names (A = 1 ... Z = 26) and the time it took them to respond to the basketball ticket offer was -.27. According to Cohen's guidelines, this is a medium strength relationship. And this for a relationship that no one (except for Carlson and Conard, I suppose) would have guessed existed at all.

Anyone who has dabbled in psychological research knows that many variables that seem like they ought to be related turn out not to be (or at only trivial levels). I can't count the number of times students in Psych 144 or Psych 42 have looked at the relationship between the number of units one is enrolled in and one's stress level and found ... bupkis.

So although the research seems well conducted and analyzed, I'm waiting to seeing some more replications--especially ones that use different methods and come from different labs. I love counterintuitive results (probably much more than the next guy), but the more counterintuitive they are, the more confirmation I want.

Stay tuned for more ... I hope.