|||

In the slim volume that he later called my disturbing little book,” Meehl reviewed the results of 20 studies that had analyzed whether clinical predictions based on the subjective impressions of trained professionals were more accurate than statistical predictions made by combining a few scores or ratings according to a rule. In a typical study, trained counselors predicted the grades of freshmen at the end of the school year. The counselors interviewed each student for forty- five minutes. They also had access to high school grades, several aptitude tests, and a four- page personal statement. The statistical algorithm used only a fraction of this information: high school grades and one aptitude test. Nevertheless, the formula was more accurate than 11 of the 14 counselors. Meehl reported generally similar results across a variety of other forecast outcomes, including violations of parole, success in pilot training, and criminal recidivism.

Not surprisingly, Meehl’s book provoked shock and disbelief among clinical psychologists, and the controversy it started has engendered a stream of research that is still flowing today, more than fifty years after its publication. The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between algorithms and humans has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented.

The range of predicted outcomes has expanded to cover medical variables such as the longevity of cancer patients, the length of hospital stays, the diagnosis of cardiac disease, and the susceptibility of babies to sudden infant death syndrome; economic measures such as the prospects of success for new businesses, the evaluation of credit risks by banks, and the future career satisfaction of workers; questions of interest to government agencies, including assessments of the suitability of foster parents, the odds of recidivism among juvenile offenders, and the likelihood of other forms of violent behavior; and miscellaneous outcomes such as the evaluation of scientific presentations, the winners of football games, and the future prices of Bordeaux wine. Each of these domains entails a significant degree of uncertainty and unpredictability. We describe them as low- validity environments.” In every case, the accuracy of experts was matched or exceeded by a simple algorithm.

Daniel Kahneman: Thinking, Fast and Slow (2011).

The algorithm wins.

    Next → Saw this the other day in Trusteeship magazine. ← Previous The statement “I had a premonition that the marriage would not last, but I was wrong” sounds odd, as does any sentence about an intuition that turned out to be false. The statement “I had a premonition that the marriage would not last, but I was wrong” sounds odd, as does any sentence about an intuition that
    Latest posts
    The Verge → Researchers detail huge hack-for-hire campaigns against environmentalists
    Conversations, cybernetics, and Theory of Mind
    → Why are we exceeding the Earth’s carrying capacity?
    IDEO U's Creative Confidence Podcast → Roger Martin, Bianca Andreescu, and systemic strategy
    Reuters → Systemic lessons from South Korea’s Patient 31
    Axle → Divide & conquer
    FSG → Can Snow Clearing Be Sexist?
    The Verge → As Lambda students speak out, the school’s debt-swapping partnership disappears from the internet
    The Talk Show → “Bring It On, Haters”, With Special Guest Ben Thompson
    Facebook → Starting the Decade by Giving You More Control Over Your Privacy
    Motherboard → Leaked Documents Expose the Secretive Market for Your Web Browsing Data
    The Verge → Google’s ads just look like search results now
    MacMillan → Interference by Sue Burke
    Systemics and design principles in support of Tiago Forte’s PARA framework
    → Microsoft wants to capture all of the carbon dioxide it’s ever emitted
    → US announces AI software export restrictions for China
    → Science Conferences Are Stuck in the Dark Ages
    → This wireless power startup says it can charge your phone using only radio waves
    → Segway’s newest self-balancing vehicle is an egg-shaped wheelchair
    → Twitter announces Bluesky: a team seeking and developing an open standard for social media
    → Elon Musk attempts to explain Twitter to normal people in court
    → TED and YouTube launch global climate initiative
    → Embracing multilingualism to enhance complexity sensitive research
    → The ‘Amazon effect’ is flooding a struggling recycling system with cardboard
    → John Kerry, Arnold Schwarzenegger wage ‘World War Zero’ on climate change
    → Combining semantic and term frequency similarities for text clustering
    → Bad RCS implementations are creating big vulnerabilities, security researchers claim
    → 2019 Tech Trends Report — The Future Today Institute
    → Medical Crowdsourcing: Harnessing the “Wisdom of the Crowd” to Solve Medical Mysteries
    → Report Launch - OPSI Primer on AI for the Public Sector
    → “Level Up”: Leveraging Skill and Engagement to Maximize Player Gameplay