The Research Production Process Edition
1. Research, Evidence, Policy and Politicians: We talk a lot around here about evidence-based policy and often about the political economy of adopting evidence-based policies. In the last faiV I featured some of the first evidence that elected officials (in this case 2000+ Brazilian mayors) are interested in evidence and will adopt policies when they are shown evidence that they work.
Far be it from me to let such encouraging news linger too long. Here's a new study on American legislators (oddly also 2000+ of them) that finds that 89% of them were uninterested in learning more about their constituents opinions even after extensive encouragement, and of those that did access the information, the legislators didn't update their beliefs about constituent opinions. Here's the NY Times Op-Ed by the study authors.
But wait, there's more! In another newly published study using Twitter data on American congresspeople, Barera et al. find that politicians follow rather than lead interest in public issues. But also that politicians are more responsive to their supporters than to general interest. Which perhaps goes some way to explaining the seeming contradictions between these two studies: American legislators are not interested in accurate data on all of their constituents' opinions, but will follow the opinions of their most vocal supporters.
2. Research Reliability: Two studies of the same population finding at least nominally opposing things published in the same week is kind of unusual, shining a brighter light on the question of research reliability than there normally is. But there have been plenty of other recent instances of the reliability of research being called into question for lots of different reasons:
* The difference between self-reported income and administrative data: the widely known finding that Americans living in extreme poverty (below $2 a day) was based on self-reported income. Re-running that analysis with administrative data that presumably does a better job of capturing access to benefits and other sources of income and wealth finds that only .11 percent of the population actually has incomes this low, and most are childless adults. Here's a Vox write-up of the findings and issues.
* A "pop" book on marriage from an academic claimed that most married women were secretly desperately unhappy. But that's because he misunderstood the survey data, believing that the code "spouse not present" meant that the husband was not in the room when the question was answered, when it really means that the spouse has moved out. Again, Vox does some good work explicating the specifics and the context: most books aren't meaningfully peer reviewed.
* But you probably should be very skeptical of any research on happiness regardless of whether it's peer reviewed because "the necessary conditions for...identification..are unlikely to ever be satisfied."
* And you should be skeptical of many papers studying the persistence of economic phenomena over time, and spatial regressions in general because of the possibility of inflated significance that is really just noise.
* You should also perhaps be skeptical of any claims based on Big 5 personality traits outside of WEIRD countries because the results are not stable across time or interviewers.
* And there are still a lot of issues with the applications of statistical techniques across the social sciences, including, for instance, the misapplication and misinterpretation of RDD designs, or conditioning on post-treatment variables (that's a paper from last year that finds 40% of experiments published in top 6 Political Science journals show evidence of doing so), or using estimated effect sizes to do ex post power calculations.
* Or this Twitter thread about a series of papers published in top medical journals that defies description, other than you really have to read it.
It's enough to make you despair.
3. The Research Production Process and Reliability: There's another aspect of research reliability in economics that doesn't get enough attention I think--how the research and publication process is set up in ways likely to create inadvertent errors. Now what follows is all speculation, but it is speculation informed by experience.
The bar for empirical research keeps going up--and that's a good thing--requiring more sophisticated experiments/analyses, with more data, bigger samples/datasets etc. But data is hard to deal with. It's messy and noisy and all sorts of other things that require a lot of time and attention that grows if not exponentially at least more than linearly as the amount and complexity of data increases. The research production process and the expectations of productivity from researchers hasn't changed though--you don't really get any more credit for a paper with a sample of 10 thousand than you do for a sample of 1000. So most of these more sophisticated projects with longer time frames and more data involve more and more collaborators and especially more and more of changing cast of RAs working on the data. And that's a recipe for inadvertent errors.
That is how I've been thinking about this recent "re-analysis" by Bedecarrats et al. of the Crepon et al. study of microcredit impact in Morocco (one of the famed 6 RCTs on microcredit impact) which incredibly ambitiously recreates the entire analysis from scratch, including rewriting all of the code into R. Bedecarrats et al. find plenty of errors in the analysis and code and question the reliability of the entire effort. I think they overstate the case, as it's not at all clear that the errors they uncover would yield a different conclusion than the original paper, but the variety of small mistakes is noteworthy.
Crepon et al. have now responded, acknowledging some errors, pushing back on others, finding some errors in the Bedecarrats et al. analysis etc. They also reiterate that the errors that are there are not sufficient to change the conclusions of the original paper.
If you follow microcredit research it's definitely worth looking at both the re-analysis and the response. But, clearly I think the more important thing is recognizing that these kinds of errors are likely common, are incredibly hard to notice, and are most likely going to become more common unless the research production process changes in some meaningful ways.
In that regard there is some good news and bad news. On the good news side, as Crepon et al. highlight in their response, J-PAL has recently launched a service for researchers where graduate students will attempt to replicate code and analysis, looking for errors. The French government has recently launched a new agency to certify the replicability of research that uses confidential administrative data, which until now hasn't been possible at all.
The bad news is that more and more data is going to become confidential as data tools allow exposing the identity of individuals involved in research. Which is going to make it even more difficult to find errors and mistakes in research.
4. Research Ethics: The reliability of research is an important question, but it's a second order one. The first order question is, "Should this research be done." There's been a lot of discussion of that question this week, as a new paper based on an experiment encouraging participation in protests in Hong Kong was released. Follow this thread for some reactions, this thread from Sheena Greitens for others (and check out her paper with co-authors on "Research in Authoritarian and Repressive Contexts"), and this thread that has some particular development econ perspectives. Spurred by the controversy, Andrajit Dube started a thread which raises the ethical questions up a level from this particular paper. And if you do a little searching there is a lot more out there.
5. Mortality: OK, we'll head in a different direction in closing. Here's an essay from Arthur Brooks that is hard but worthwhile reading: Your Professional Decline is Coming Much Sooner than You Think.
L-IFT, a diaries research firm, is a proud sponsor of the faiV. See our video on diaries research with 800 women microentrepreneurs in Myanmar.
This came across my Twitter feed yesterday and I thought it would be a nice diversion from all of the more serious topics above. It's a murmuration of starlings in Costa Brava, photographed by Daniel Biber. Source.
