The credibility revolution seems like a dream come true for sociology. Sociologists want to know how the social world works. They want to know what causes what and how important social effects and phenomena are produced. Even if they lean activist, it should be imperative for sociologists to identify social causes and effects – precisely because they want to change the social world (for the better). That’s because you can’t design interventions and hope to “cure” social ills with mere correlational knowledge, where variables X and Y are just associated with each other but not necessarily (or even likely) causally connected.
If you as a sociologist find out that, say, globalization is correlated with higher poverty, your ostensibly logical advice that we should stop globalization to prevent poverty from increasing could in fact be highly illogical. Only in the case where globalization causes higher poverty would preventing further globalization stop poverty from rising. But knowledge of the two phenomena being merely correlated doesn’t tell you that. With correlations, you have to fear the dreaded selection bias. For instance, it could be that countries with higher poverty are, for that very reason, being targeted by the international community to help them and infuse them with capital (or, less benevolently, to use their cheap labor), which means that they suddenly become more globalized.
In that case, poverty and globalization surely go together, but not at all because globalization is causing poverty. If you tried preventing globalization based on this correlational knowledge, you’d be doing something bad or at best something irrelevant (i.e., definitely not something that would stop poverty). Selection bias and associated methodological horrors – stemming from endogeneity – are pervasive in observational (social and medical) science.
As Scott Cunningham explains at length:
Optimization Makes Everything Endogenous
Economic theory tells us we should be suspicious of correlations found in observational data. In observational data, correlations are almost certainly not reflecting a causal relationship because the variables were endogenously chosen by people who were making decisions they thought were best. In pursuing some goal while facing constraints, they chose certain things that created a spurious correlation with other things. And we see this problem reflected in the potential outcomes model itself: a correlation, in order to be a measure of a causal effect, must be based on a choice that was made independent of the potential outcomes under consideration. Yet if the person is making some choice based on what she thinks is best, then it necessarily is based on potential outcomes, and the correlation does not remotely satisfy the conditions we need in order to say it is causal. To put it as bluntly as I can, economic theory says choices are endogenous, and therefore since they are, the correlations between those choices and outcomes in the aggregate will rarely, if ever, represent a causal effect.
Enter: the credibility revolution
Statisticians and econometricians have been coming up with clever methods and strategies of using observational data (and, of course, actual randomized experiments or RCTs) to plausibly go beyond correlations for decades now. And not only is uncovering causal effects with observational data nowadays possible, it can work even when – as is very likely in certain key sociological domains – the sample size is small and you’re interested in (historical) case studies.
You know the story. Instrumental variables (IVs), difference-in-differences (DID) designs, the synthetic control method (SCM), regression discontinuity (RD), and so on allow us under certain circumstances to better think about counterfactuals and infer causal relationships. Interested in the effects of international sanctions on the size of the middle class in Iran? That’s a perfect historical, case-study, sociological question that requires causal inference and isn’t amenable to standard statistics that rely on sampling and large datasets. How about the consequences of CIA-sponsored regime change in Latin America? Or the long-term effects of sectarian politics in Lebanon? What about how smartphone usage affects well-being and learning among children and adolescents in Norway? Sociologists, rejoice, your time has come!
As always, there are giant pitfalls that have to be avoided when handling these methods and coming up with causal designs. As typically practiced, causal inference based on IVs can be quite problematic. SCM can definitely be fragile and abused, as noted even here on Substack by Nicholas Decker. This is especially the case if you’re not using the latest augmentations or not doing anything to prevent p-hacking when constructing the control.1 And just sticking a DID label on your garbage-can regression doesn’t solve anything, obviously.
(A recent manuscript where I acted as one of the peer reviewers did just that. No identification strategy, no operationalization of treatment, no discussion and justification of the method’s assumptions, like parallel trends, no testing for anticipatory effects and other placebo effects, no nothing. Just an ordinary panel regression that the authors called a “DID” analysis and relied on to refer to their results as “causal.”)
But these issues are not insurmountable. Using best-practice steps and checklists helps prevent some of the abuses (whether intentional or not). Running placebo and falsification tests, alongside a variety of robustness checks, boosts confidence. Perhaps most importantly, possessing deep institutional knowledge of the social setting you’re working with is a key leverage in setting up your causal-inference design. Such (quite often qualitative) knowledge is something we sociologists pride ourselves on! And it’s one of the things that is very useful when thinking about the counterfactual in your causal analysis.
In short, I think the credibility revolution is at least setting social science on the right track, even if we still have a long way to go in practice.
Sociologists don’t seem to care much for it
What is profoundly troubling, or at least saddening, to me is that despite how important this development has been and how neatly it goes together with sociology (i.e., its hunt for causes of social effects and its ostensibly deep institutional knowledge), sociologists mostly seem not to have noticed it or cared about it.
There’s not much data on the issue so we should be careful here. I don’t want to rely on my personal impression and anecdotes, so let’s go with something at least slightly more systematic. Jukka Savolainen sampled three flagship journals in each of the four big social science fields (economics, political science, sociology, and psychology), selecting only empirical studies. He focused on the most recently published issue (observed in spring 2025). For his analysis, he applied a “credibility-revolution” coding rule: that is, a paper counts only if it contains at least one clean counterfactual estimate from an RCT/field experiment or a quasi-experimental method like DID, RD, IV, and so on.
The sample size is nothing impressive (so, again, caution about generalization is advised), but the results are quite damning. As expected, he finds that economics is awash in counterfactual, causal designs (72.6% of the empirical articles) and political science isn’t far behind (60.7%). Even psychology lands at 50%. Sociology’s flagship-journal share, by contrast, is below 10%! Sociology is the clear outlier, seemingly almost completely oblivious to the credibility revolution.
I can’t stress enough how weird and unnecessary, even ironic, this is. In fact, Kevin Grier made a very similar observation about Austrian economics some time ago. For the most part, with important exceptions, Austrian economists simply don’t care about the credibility revolution. Grier notes that’s very strange because in reality, Austrians are “uniquely placed to be effective practitioners of causal inference techniques on observational data.” Why? Well, because, as Grier puts it:
(1) all the models have identifying assumptions (e.g., no spillovers and parallel trends) that are best addressed by an exposition of the institutional/economic/historical milieu in place before and after the treatment under study;
(2) determination of external validity also requires detailed institutional and historical knowledge;
and (3) researchers often want to know the mechanisms producing the reduced form result that comes out of most causal inference studies. Here again, institutional and historical learning is crucial.
I’d add that, if anything, sociologists (rather than Austrians) are uniquely placed to exploit this development. That’s because they possess the same thick knowledge of institutional details of individual cases, are just as interested in adducing mechanisms, and have also (in contrast to the Austrians) always been both interested and trained in quantitative empirical testing of their hypotheses.
Alas, currently, they seem uninterested in the prospect.
Like he himself says, most of the issues or objections brought up in his list are quite easily and routinely solved. I think he also underestimates how much augmented synthetic control helps and what synthetic difference-in-differences can do for us. He correctly concludes by saying “I think synthetic controls do have extraordinary potential. I entirely agree with people like Susan Athey and Guido Imbens when they describe it as one of the most important innovations in econometrics in this century.”
One downstream consequence of this, I think, is that moral certainty starts doing the work that causal inference is supposed to do. Once a narrative feels righteous enough, the pressure to establish counterfactuals disappears. At that point, disagreement is treated as immorality rather than as a demand for better evidence. That shift feels less like a technical failure and more like an epistemic one.
Interesting post. Indeed, the credibility revolution should be transforming all social sciences, not just economics and political science. To what extent do you suspect sociology’s lack of take up to be driven by: i) relatively poorer training in statistics and probability theory and ii) relatively less emphasis on falsifiable theorizing? I think that the former is related to a suspicion of quantitative methods, and their ability to “understand complex meaning”, while the later lends itself to confirmatory research practices. A greater appreciation for the relative strengths of quant methods, and a greater willingness to empirically falsify a hypothesis, are both necessary for the credibility revolution to reach sociology (in my opinion).