This is the second of three posts addressing the standard critiques of RCTs. In the last post I addressed the External Validity Critique. In this post I’ll take up the Transcendental Significance Critique—or put a different way, the “It doesn’t matter anyway” critique. In the final post in the series, I’ll discuss some of the problems of interpretation and implementation of findings from RCTs.
The Transcendental Significance Critiques takes several different forms. One is evident in my interaction with Eric Meade on the Stanford Social Innovation Review Blog. This version suggests that RCTs don’t effectively shed light on a grand epistemological view of poverty and social change (this is a different from a critique about theory-less RCTs, a different topic entirely). Another version suggests that RCTs are irrelevant because they cannot be used to measure what really matters—which isn’t foreign aid or charitable programs. In Angus Deaton’s version of Transcendental Significance Critique focuses on national policy and broad development initiatives which can’t be field tested. Philip Auerswald’s version keys on entrepreneurship and economic dynamism which he believes are the real drivers of development and change. Finally, there is a version of the critique which focuses on the static nature of any field experiment. The results of an RCT only tell you about a particular moment in time, and usually well after that moment in time has passed. This critique argues that the world is so dynamic that moment in time snapshots are not useful.
The Transcendental Significance Critique in most of its forms is valid. There are many factors that are likely to have a great deal of influence on development and poverty alleviation that are not subject to RCTs—at least not at reasonable cost and time frames (though do read David Roodman on this). Programs to meet the needs of the poor, whether government-funded, like conditional cash transfers or privately-funded like microcredit or well-boring, are dwarfed by the potential impact of general economic growth and macro policy.
But, I would argue, that does not invalidate the effort to make programs that benefit the poor more effective even if the benefits are only on the margin. Let me explain using a personal story. My 5 year old son suffers from a rare genetic syndrome that makes many of his daily activities just a bit more difficult than they are for the average person, but most heartbreakingly, will render him blind by the time he reaches his 20s, if not sooner.
There have been efforts for centuries to help the blind navigate the world more easily and limit the impact of their disability. My son is learning Braille and how to use a cane—the most well-known fruits of that work. But there is also cutting edge research going on around the world to develop tools to help even more. For instance, check out this hacked XBOX Kinect which provides tactile feedback to enable the blind to navigate an unfamiliar environment unaided.
But none of these things—not Braille, not canes, not hacked Kinects—truly solve the problem. None of them will restore my son’s vision once it’s gone. Thankfully there are also people working on that front—various forms of gene therapy that would replace his misfiring genes and repair his deteriorating retinas (if you need something to brighten your day, I highly recommend these videos) or replace them with bionic eyes.
Unfortunately, gene therapy is turning out to be much harder and more complex than researchers imagined. In the late ‘80s many researchers believed that within 15 years, gene therapies would be common. Today there are only a couple that have been approved. While the concept of gene therapy works well in theory, it is very hard to accomplish in practice.
Does all of this lead me to think that those guys hacking the Kinect in the Netherlands are wasting their time? That their efforts and funding should all be turned to gene therapies or electronic retinas? That their work is a luxury that’s too expensive and doesn’t really matter anyway?
Of course not. My fervent desire is for the “cure” to be discovered. But I don’t turn my nose up at research that won’t “fix” my son’s eyes—I equally fervently support the research that will limit the effect blindness has on his enjoyment of life. I do expect that any efforts will be tested so that new efforts to “help” him really will help him.
The hopes for clear advice on macroeconomic policy for development or for promoting the kind of entrepreneurship that makes a difference for development are a lot like gene therapy. In theory we should be able to figure out these problems, but in practice they are proving very difficult to come to satisfying and useful answers. While pro-poor policies and programs may not ultimately be the solution to poverty, there is no question that they can improve the lives of the poor in the near term if we invest in high quality research like RCTs. It’s even possible that the marginal improvements in the lives of the poor that we can influence by improving pro-poor programs and policies will have an enabling or amplifying effect on those things that the Transcendental Significance critics claim “really matter.” Is it really better to spend our energy and efforts only on the “big” things that we may not be able to affect than on the “small” things that we can affect?
I don’t think so. That’s why I believe that RCTs do really matter. They may be a luxury but they are a luxury worth investing in.
Timothy Ogden is an executive partner at Sona Partners, the editor in chief of Philanthropy Action, and co-author of Toyota Under Fire. He also blogs at HBR and SSIR.