Wednesday, February 16, 2011

Clinical Prediction Fails Again

I just gave the first Psych 144 exam of the semester. Before I graded them, though, I tried to predict everyone's score based on my experience talking with them, observing them work on activities, and so on. (I did not know how anyone had scored on quizzes or assignments because my TAs grade those.) My intuition told me that my predictions would be pretty good and I was expecting a correlation between my predictions and the actual scores of at least +.50 and probably higher.

The reality, though, was that the correlation was a measly +.17. (I won't show the scatterplot because people might be able to tell from it how I had predicted they would do. But the pattern in the data is barely detectable in it.)

To put things in perspective, I also computed the correlation between the order in which students finished the exam and their scores. It was -.40. (The negative correlation just means that students with lower ranks tended to score higher.) In other words, simply looking at the order in which students finished provided much better predictions than I did using my "professional judgment" based on experience.

[Consider also that this was on a day with horrendous traffic because of the "Get Motivated" event at the Save Mart Center, which I'm sure motivated a lot of complaints to the University Administration. Anyway, this made many students late and affected the order in which they finished. The correlation would probably have been stronger than -.40 otherwise.]

These results are actually typical of tons of research on "clinical prediction" (prediction done based on experience and intuition) and "actuarial prediction" (prediction based on a simple rule usually derived statistically from past data ... but in this case just pulled out of thin air). This research shows that although psychologists, doctors, teachers, and others tend to be quite confident in their clinical predictions, actuarial predictions are always more accurate overall.

One of the best books about psychology (in my opinion, of course) is House of Cards by Robyn Dawes, in which he discusses the problems with clinical prediction and their implications for clinical psychology. I strongly recommend it.

Dawes, by the way, is one of my own intellectual heroes. Unfortunately, he died just recently. (Here is an obituary for him.) Although I had chances to meet him, I never had the nerve to walk up and introduce myself. I definitely wish I had.

No comments: