Having a high pastorate providing divinely inspired advice for the direction of a society would seem a theme worthy of a great dystopian novel, whether your tastes run to the likes of Margaret Atwood or Robert Heinlein.
However, it seems we’ve been living through one for years. Not run by a literal religious sect, but by a figurative one: economists. All the research, ideas, and suggestions they churn out for peer-reviewed papers taken up by pundits, political leaders, and the press as proof of what should happen in the future.
Unfortunately, according to research at — the irony is too much for fiction — the Federal Reserve, the results more often than not can’t be reproduced. One of the basic requirements of scientific inquiry is that others should be able to replicate one’s results if they are generally true and not some product of specious accident. That fundamental apparently has left the room for an extended lunch.
The paper, first published in 2015 through the Federal Reserve Board, should have rung a deeper and more penetrating reverberation at the time. Even the title was brutal: Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ”Usually Not”.
It was a staff working paper. As such, it was one of those “preliminary materials circulated to stimulate discussion and critical comment” that “do not indicate concurrence by other members of the research staff or the Board of Governors.”
Understandable that an organization based on the perception and wisdom of economic research might not support something that suggested, from a scientific view, much of the work could be hooey.
The authors, Andrew Chang, currently a principal economist at the Board of Governors of the Federal Reserve System, and Phillip Li, a senior financial economist at the Federal Deposit Insurance Corporation, tried to replicate 67 papers from 13 “well-regarded” economics journals.
First trouble, some of the journals required data and code and others didn’t. That left them obtaining data and code for 29 out of 35 papers (83%) that were required to provide them and 11 out of 26 papers (42%) that were not required to provide them. There were 6 papers that used confidential data that was unavailable. (Adding the 35 required to provide data, 26 that weren’t, and 6 that used confidential data gets the total 67.)
Excluding the 6 papers with confidential data and 2 more that used software unavailable to the co-authors, they were able to replicate 29 out of 59 papers, or 49%, with assistance from the authors.
“Because we are able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is usually not replicable,” they wrote.
During the recent jump in inflation, given the number of name economists who kept pointing to the Phillips Curve — a proven unreliable correlation — as why the country needed higher unemployment to battle rising prices, the lack of concern over replicable research shouldn’t be surprising.
What surfaced this roughly nine-year-old paper was an article in The Economist about the rise and fall of “Freakonomics,” the book — and eventually radio show, podcast, and what have you — that was supposed to illuminate “the hidden side of everything.”
Specifically, the article touched on a touchy chapter claiming that the 1973 legalization of abortion led to a fall of crime in the 1990s because of the many future criminals who never came to being.
“It was a classic of the clever-dick genre: an unflinching social scientist using data to come to a counterintuitive conclusion, and not shying away from offence,” The Economist wrote. “It was, however, wrong.” A coding error and the use of total arrests, rather than arrest rate to see amount of crime per population, undermined the claims. Other researchers eventually noted that the beginning of the drop was among women, not men, and that no-fault divorce “may have played a bigger role.”
Most of the article is about the entangled issues of supporters and detractors and said that “subsequent papers often overturned results, even if, as in the case of those popularised [in English spelling] by Freakonomics, they had an afterlife as cocktail-party anecdotes.”
“The centuries-old questions of economics are as interesting as they always were,” the article ended. “The tools to investigate them remain a work in progress.”
If the progress is so rough, perhaps there should be less obeyance and more abeyance in heeding what economics are pretty certain they know, until they don’t.