In honor of Angus Deaton's Nobel prize announcement, below is an excerpt from the forthcoming book Experimental Conversations, to be published by MIT Press in 2016. The book collects interviews with academic and policy leaders on the use of randomized evaluations and field experiments in development economics. To be notified when the book is released, please sign up here.
The full, annotated interview with Angus Deaton is available on Medium.
Timothy Ogden: Another of the common critiques of the RCT movement is a lack of a theory of policy change.
Angus Deaton: I think that’s a very complicated thing. These things are slow often, but there is a big political element and there should be. Something I read the other day that I didn’t know, David Greenberg and Mark Shroder, who have a book, The Digest of Social Experiments, claim that 75 percent of the experiments they looked at in 1999, of which there were hundreds, is an experiment done by rich people on poor people. Since then, there have been many more experiments, relatively, launched in the developing world, so that percentage can only have gotten worse. [vi] I find that very troubling.
If the implicit theory of policy change underlying RCTs is paternalism, which is what I fear, I’m very much against it.
I think policy change very much depends on the context. I don’t know if you’ve read Judy Gueron’s book.[vii] I learned a lot from that. What do these MDRC things do? They’ve gone on and on and continue to this day. Many academic economists were involved in the early days but much less since but MDRC and Abt and Mathematica and so on have gone on doing these experiments ever since. For the Federal government, state governments, and some in Canada. So I’m kind of curious about how they function in the policy space.
I don’t think the results from these experiments have had much of an impact on academic knowledge, but that may be wrong. I don’t know. I think what the experiments did was to settle disputes between competing political views. There would be a new administration and they would say, “These policies should all be abolished,” or, “If we make people go to work before we give them any welfare that will cause them to earn their own incomes and it will reduce costs for the government,” for example. The interesting thing is that in the US such arguments have to be costed by the CBO which actually has to estimate if the financial projections that come out of those proposed policy changes make any sense. When the Reagan people came in, they were not keen on doing any experiments at all but when the CBO didn’t agree with their estimates, they became supporters of experiments because they believed it would show that they were right. And sometimes indeed they were, at least as far as RCTs can tell.
Those experiments are mostly about what policy changes did to the budgets of state and federal authorities. They have a case load, and they often care less about the well-being of poor people than they care about the state budget. An RCT is good for that because it gives you the average cost and the average in that context is exactly what you want to know. It resolves the dispute. But that average is not generally useful elsewhere, at least without understanding the mechanisms. And MDRC wrestled with that problem of finding mechanisms from the very beginning but they never resolved it. They thought that by going into the details they could find mechanisms that would generalize or transport and they never managed to do that. You can’t do that with RCTs. You’ve got to combine them with theory and observational data so you’re right back where you were.
But before you need a theory of policy change you need a theory of transportability. Meaning, it works here, what arguments do you have that it works there? And it often seems that those running RCTs simply assume that these numbers will apply with little discussion of how to move the results from one location to another.