to Miami | Main | the earmarking/lobbying connection

January 05, 2006

the limits of formal experimentation

(Miami, FL) In a genuine experiment, you somehow interact with a randomly selected group of people but leave everyone else in your sample alone. You assess everyone at the end of the experiment, assigning them scores based on a survey, test, or other instrument that measures what you care about. If the average score of the people whom you attempted to influence is higher than those in the control group, then what you did worked. There are some complications, but this is the essence of formal experimentation.

It has some big advantages ...

  • It actually measures causality.
  • It creates accountability by forcing people who run social programs to demonstrate results.
  • It shows what is cost-effective. This is crucial because resources are limited, and some well-meaning programs are actually counter-productive.
  • It is a relatively simple process, much easier to master than statistical modeling. It is well within the capacity of high school students and community groups, if they can overcome bureaucratic obstacles. (Professionals often use multivariate models to approximate the results of experiments when no actual experiments have been conducted. This approach requires more technical expertise than experimentation, but it is less reliable.)
  • Therefore, I am drawn to arguments that we ought to become a much more experimental society. The funding of true field experiments should become a major role of government, as we try to design programs that can actually address the challenges of the current era.

    However, there are limits. Experimentation sometimes raises ethical issues regarding the treatment of research subjects. Experiments can only test a few interventions at a time, yet often we want to explore the possible impact of many factors, not knowing which are most important. Some important factors are hard to measure; they require qualitative research (which I strongly support).

    Finally, experiments test the impact of general, repeatable strategies and approaches. But many of the most important decisions we make are highly contextual and uncategorizable. For example, consider this espisode from a recent conference on psychotherapy, as reported by Benedict Carey in the New York Times on Dec. 27:

    Many therapists at the conference said that if the field did not incorporate more scientifically testable principles, its future was bleak.

    Using vague, unstandardized methods to assist troubled clients ''should be prosecutable'' in some cases, said Dr. Marsha Linehan of the University of Washington, who has developed a well-studied method of treating suicidal patients.

    Yet it was also apparent in several demonstrations of the spellbinding thing itself -- artful psychotherapy -- that some things will be difficult, if not impossible, to standardize.

    Dr. Donald Meichenbaum, research director of the Melissa Institute for Violence Prevention and Treatment in Miami, showed a film of the first session he conducted with a woman who was suicidal months after witnessing her boyfriend die in a traffic accident. After gently prompting her to talk about the accident, Dr. Meichenbaum then zeroed in on something he had noticed when the woman entered his office: she was clutching a cassette tape.

    He asked about the tape and learned that it was a recording of her late boyfriend's voice, expressing love for her. ''I play it over and over, and it makes me so depressed,'' said the woman, in a tiny voice.

    And here Dr. Meichenbaum stopped the film and addressed the audience.

    ''The tape!'' he said. ''When during the session do you go for the cassette tape? What do you do with the tape?''

    For several long moments not a creature stirred, not even a laptop mouse. This community of therapists was now trying to save a soul, a person who was alone and did not want to live. What to do with the tape?

    I don't see any way to use experiments to answer this crucial question. Similar issues, of course, arise in teaching and in the management of public institutions.

    Posted by peterlevine at January 5, 2006 07:21 AM


    There are two principle limits to experimentation: 1) feasibility and 2) external validity. Your example from the NYT fits into the first category, as a wide-range of problems don't have proper (or merely cost-effective) ways to create control and treatment groups. Sometimes overlooked though is the second point, which has to do with often contrived settings of the laboratory.

    Perhaps social science could do a little more true experimental research, but there are good reasons that it represents only a small (5-10% perhaps?) of all empirical social science research. Otherwise, how can we even approximately learn how to get that tape from the clutches of the suicidal patient?

    Posted by: Michael Weiksner [TypeKey Profile Page] at January 11, 2006 05:35 PM

    Post a comment

    This blog is under attack from comment spammers, who are causing a problem for the server. I believe I can block them by upgrading to a recent version of MoveableType. However, I do not have time to do that until late December. Therefore, I have temporarily disabled comments. Please feel free to email me feedback at

    Site Meter