1. Economics? What Is It Good For?: It's hard to spend any time paying attention to methodological and disciplinary debates without thinking of the Planck/Samuelson dictum about science advancing via funerals. Here, I'm thinking of attitudes toward the value of field experiments specifically and the "credibility revolution" generally. Christopher Ruhm recently gave a speech, in paper form here, about the "credibly-answered unimportant questions" vs "plausibly-but-uncertainly-answered important questions" debate. I found it helpful because it makes the hollowness of this concern more evident than usual, but you'll have to wait on the book chapter I'm procrastinating on to read why. Noah Smith has a more charitable take on Ruhm's speech, with the added important note that one of the big problems of the field is that outsiders don't understand the difference at all.
On the credibility side of things, there are issues beyond just the identification strategy. Here's an interview with Ted Miguel on transparency and reproducibility, a neglected part of the credibility revolution as far as I'm concerned. David Roodman has resurfaced with two new papers doing the hard work of reproducing results. He looks at Bleakley's study of the effects of hookworm elimination in the US and of malaria control in the Americas, questioning the result of the first, but largely upholding the result of the second.
But there's yet another dimension of credibility that I feel like is even more neglected, hearkening back to Paul Romer's mathiness paper: the comprehensibility of methods and tools. Here's a recent example: Declare Design has a lengthy discussion of whether and when to cluster standard errors, inspired by questions posed by David McKenzie and Chris Blattman. It's great. But is anyone else concerned about how few people actually understand the statistical methods we rely on? And that problem is going to get worse, as more and more machine learning and AI techniques come to the fore, techniques that perhaps even fewer understand. And the people that do understand them often don't understand causal inference or the philosophical issues around such basic concepts as fairness.
I guess, therefore, in fairness I should point out that apparently economics is good for sports, specifically the NFL (at last), and it is good for showing that the Planck/Samuelson dictum is true.
2. A Clash of Civilizations: Part of the curious thing about the way the RCT debates in economics evolved is the frequent citing of the use of RCTs in medicine as justification for their use in economics. It's curious because seemingly the understanding of causal inference methods in medicine isn't great. Here's a piece from JAMA (trigger warning: it calls RCTs the gold standard) on why you shouldn't take people out of your treatment group and put them into your control group because the treatment didn't work for them. It's not quite that bad, but still. Here's a thread from Amitabh Chandra on that paper and the general lack of causal inference understanding in medicine.
And here is a fascinating piece of work about how causal claims in health research get steadily ratcheted up. The authors looked at the 50 most shared journal articles about the health effects of exposure to something, finding "that only 6% of studies exhibited strong causal inference, but that 20% of academic authors in this sample used language strongly implying causality." And then the general news media further ratcheted up the causal claims.
I include that as important background to the clash of civilizations that happened recently when Jennifer Doleac, Anita Mukherjee and Molly Schnell wrote about the causal effects of harm reduction strategies related to opioid addiction, reviewing the literature and especially their paper on the impact of naloxone distribution. They find that naloxone access reduces short-term mortality but increases long-term mortality. That doesn't sit well with a wide variety of people outside economics. This is one of the tamer reactions from outside economics (trigger warning: it also refers to RCTs as the gold standard), tamer in the sense that it actually attempts to grapple a bit with the issues. But it ultimately settles on a version of the trope that "we already know the answer, so your causal inference sucks" and "Here's a study of a different intervention that works, so your causal inference sucks." You have to admire (well, you don't, but I do) Doleac for continuing to wade into controversial topics where there are people with very strong priors such as whether bail-setting algorithms might in fact be fairer than judges.
Public Health and Medicine aren't the only areas where economics clashes with other disciplines. Perhaps that has something to do with how insular economics publishing is. Tying all this together, here's a thread from Jake Vigdor about economic publishing insularity (See Graphic of the Week below) linking to this very cool set of visualizations about cross-disciplinary references in academic journals. Suffice it to say Econ is not doing well at being noticed outside of Econ journals. Perhaps the Doleac et al paper may make a dent in the public health journals.
3. Impact, Scale, Policy, Oh My: One of the critiques of the experimental approach is that it necessarily must focus on relatively short-term effects, when from a policy perspective we should care much more about long-term outcomes. Here's a new paper from Bouguen, Huang, Kremer and Miguel on the possibility of long-term follow-up of RCTs finding that many do lend themselves to possible long-term follow-up, and offer advice on how to make more studies amenable.
Another critique is the necessarily small scale of RCTs and that the results tell us little about would happen if the intervention was scaled up (and the historical record has lots of examples of interventions where the effects faded out when scaled up). Now, the "necessarily" part has been pretty conclusively debunked, it is absolutely true that most experiments are small scale (cf. that same link). Yale has a new program specifically to study issues of scaling up: the Research Initiative on Innovation and Scale, or Y-RISE. They recently held their first conference on external validity, with lots of tweets. Here's a Vox summary of a conversation with faculty director Mushfiq Mobarak. This is obviously written for a general audience, but I have to point out that everything in this discussion can be found in critiques of RCTs by people like Lant Pritchett written years ago. I expect this to feature in an updated version of Lant's presentation, "The RCT Debate is Over. I won."
Now the thing that is obscured in Lant's framing is that the main argument for "I won" is that the randomistas are doing nearly everything that he advocated. Studying how to effectively scale-up is an example. Another LantRant is how impact evaluations are often essentially adversarial and outside the chain of policy-making. That's something the randomistas are doing something about as well. Here's a new J-PAL report on their partnerships with Latin American governments to both conduct and use impact evaluations as part of policy-making.
4. My Wish List: Seema Jayachandran is taking a term as an associate editor at the Journal of Economic Perspectives, and is wondering what topics in development you would like to see covered in JEP. Here's my wish list:
1) An update on microcredit impact since the Banerjee Karlan Zinman 2015, with a focus on heterogeneity, intrahousehold decisions, general equilibrium, labor and other market effects and mediators. (Yes, that's exactly the topic of the first faiVLive. If you haven't seen it, you should click here. Especially if you're Seema or any of the other JEP editors).
2) A synthesis of the low-income household savings literature, including studies in high-income countries, from the last decade.
3) David McKenzie writing a review of his dozen(s) of papers on micro- and small firms, training, formalization, etc.
4) A paper that revisits behavioral econ/finance ideas, in the wake of much of the social psychology underpinnings getting shaky, (e.g. every sort of reminder you can imagine fails to have an effect on medication adherence).
5) Someone reflecting on the curious case of the "no short-term effect but big long-term effect" (e.g. deworming, MtO, now HeadStart), and the "big short-term but no long-term effect" (e.g. Blattman, Fiala) phenomena
What's on your wish list?
Perhaps, though, your wish list turns more toward who should win the next few Nobel prizes. If so, you can submit an essay to Econ Journal Watch.
5. To Take Your Mind Off Things: And finally, here are a few odds and ends that have nothing to do with the faiV's usual topics to use for some R&R over the holidays. How about What termites can teach us? Or What if the placebo effect isn't a trick? (OK, that one isn't so far off normal faiV topics). How the Math Men Overthrew the Mad Men? Now I'm realizing that perhaps my interests aren't as varied as I like to think.
Well, this is truly a breakaway. Here's a book that you should buy and read over teh holidays, and it's not a big commitment because it's a set of essays, so you won't add to your guilt pile. Impossible Owls by Brian Phillips.
Here's a portion of one of those visualizations of Econ journals links, or lack thereof, to other disciplines. This is only part of of only one of the visualizations but you can see the self-referential nature of citations. Source: I'm not really sure what to call it.
