A Piketty problem?

THIS morning, the Financial Times leads with a striking allegation: “Capital in the Twenty-First Century”, the bestselling analysis of inequality by economist Thomas Piketty, is fundamentally flawed thanks to errors in the data backing the book. The story is based on work done by Chris Giles, economics editor of the paper. He writes that his interest in the data’s veracity was piqued by an apparent large disparity in the figure for the concentration of wealth ownership in Britain used by Mr Piketty and that reported by Britain’s Office for National Statistics. In a companion blog post, Mr Giles lays out the charges and concludes, “The conclusions of Capital in the 21st century do not appear to be backed by the book’s own sources.” A damning statement, if true.

Mr Giles’s analysis is impressive, and one certainly hopes that further work by Mr Giles, Mr Piketty or others will clarify whether mistakes have been made, how they came to be introduced and what their effects are. Based on the information Mr Giles has provided so far, however, the analysis does not seem to support many of the allegations made by the FT, or the conclusion that the book’s argument is wrong.

There are four important questions raised by the FT’s work. First, which data are wrong? Second, how did errors in the work, if they are errors, come to be introduced? Third, how do the errors affect the specific points made in the relevant chapters? And fourth, how do the errors affect the fundamental conclusions of the book?

Start with the first question. “Capital” is a sprawling book based on an enormous pile of reference material and data. Much of the data was collected by Mr Piketty and other economists in a series of published papers that have since been used to create the World Top Incomes Database. None of this work appears to be at issue. Rather, Mr Giles focuses on wealth inequality, to which Mr Piketty turns in Chapter 10 of his book. Mr Piketty has not published nearly as much research on the question of wealth inequality, and it seems that much of the analysis in Chapter 10 was done specifically for the book, based on others’ research. Mr Piketty’s wealth-inequality analysis certainly matters as a component of the book’s argument, but it is not accurate to say, as Mr Giles does, that the results in Chapter 10 constitute the “central theme” of the book.

Are the data wrong? Mr Giles identifies discrepancies between source material cited by Mr Piketty and the figures that appear in the book. He identifies cases in which Mr Piketty appears to have chosen to use data from one source when another would have made more sense. Further, the calculations in Mr Piketty’s spreadsheets (which have been available online since the book’s publication) seem to include adjustments in the data that are not adequately explained, and some figures for which Mr Giles cannot find a documented source. Finally, Mr Piketty has made choices concerning weighting of data used in averages, and assigning of data from one year (1935, for example) to another (1930) when such assignments seem unnecessary or inadvisable.

It is not easy to verify whether the data are in fact wrong. There are a couple of cases where it looks as though a transcription error has been made (where Mr Piketty may have grabbed a figure from one line in a chart rather than another), but one cannot be certain. There are other examples where adjustments have been made to the data that are difficult to parse out. But as Mr Giles allows, the data sources underlying the book are often sketchy and rarely perfectly comparable; any analysis would require tweaks to the data (though one would, of course, like those tweaks to be explained and documented in detail). As economist Justin Wolfers writes in the New York Times, “it’s not yet clear whether the cause is obvious errors as pointed out by the newspaper, or judgment calls where perhaps the professional economist deserves the benefit of the doubt.”

The second question concerns how such errors came to be made, if errors they are. As mentioned, transcription mistakes are one possibility. But while some of the data and adjustments in the spreadsheets lack adequate documentation, Mr Giles does not have the evidence to justify the implication that figures are drawn “from thin air”. Data fabrication is a serious charge to make, and I am surprised Mr Giles would allege it without clearer proof.

The third question is the one many readers will find most relevant: how do the data change the picture? Mr Giles examines the figures on wealth concentration in four countries: Britain, France, Sweden and America, as well as the European average. For France and Sweden the picture is most clear: there are some differences in Mr Giles’s work and Mr Piketty’s but the trends are basically unchanged. For America, the outcome of the analysis is relatively muddy (based in part on the fact that the source data themselves are harder to parse, according to Mr Giles). The trend for the top 1% share does not seem to be affected. For the top 10% it is harder to say; here is a chart Mr Giles produces:
Mr Giles reckons data presented by other economists show a gentler rise in inequality, which is a fair criticism, but it is not obvious that Mr Piketty has made any glaring error. The British case is the hardest to gauge. Here is another of Mr Giles’s charts:

Mr Giles writes that the gap “appears to be the result of swapping between data sources, not following the source notes, misinterpreting the more recent data and exaggerating increases in wealth inequality.” The FT has published a response letter from Mr Piketty (who was told of the analysis on Thursday) but it does not address specific allegations. It is a challenge to understand what might have happened without more explanation from Mr Piketty. For the moment, Mr Giles’s work suggests that a mistake has been made in the Britain analysis, and that as a result of that mistake the level of wealth inequality is overstated. It is less clear that Mr Piketty’s analysis has overstated the recent trend toward rising inequality, as Mr Giles suggests.

Finally, Mr Giles argues that if one uses a Britain series of his own construction and uses a Europe average weighted by population rather than a simple average, the trend toward rising wealth inequality across Europe flattens out. Without more clarity on the Britain question it is hard to judge this claim. It is not obvious that weighting by population is a better choice than alternatives (like weighting by GDP). A broader point concerns whether an average of Britain, France, and Sweden should be represented as “Europe”, but that of course is the way Mr Piketty presented the data in the first place.

The fourth question is whether the book’s conclusions are called into question by Mr Giles’s analysis. If the work that has been presented by Mr Giles represents the full extent of the problems, then the answer is a definitive no, for three reasons. First, the book rests on much more than wealth-inequality figures. Second, the differences in the wealth-inequality figures are, with the exception of Britain, too minor to alter the picture. And third, as Mr Piketty notes in his response, Chapter 10 is not the only analysis of wealth inequality out there, and forthcoming work by other economists (some conclusions of which can be seen here) suggests that Mr Piketty’s figures actually understate the true extent of growth in the concentration of wealth.

However, given the questions that have been raised it would be inappropriate to say anything definitive. One hopes there will be an additional response from Mr Piketty. There will no doubt be efforts by other scholars to dig into Mr Piketty’s figures; Scott Winship, a scholar at the Manhattan Institute who disputes Mr Piketty’s overarching narrative about inequality wrote on Twitter last night:

I’ve spent time with Piketty U.S. wealth ineq[uality] spreadsheet and LOTS of time with his income data. He’s not up to funny business.

And of course, it will take future research to show whether the broad strokes of Mr Piketty’s book are correct or not. But that was true before the FT analysis was published as well.

Now, the academic debate is a different thing from the judgment reached in the court of public opinion. There was an outbreak of gloating across the wires the moment the Financial Times story went live. The book has plenty of critics (many of which never spent much time wrestling with the book’s arguments in the first place), and many of which reached gleefully for word that Mr Piketty’s work might not be perfect. One suspects that in a public back-and-forth that has often failed to hew particularly closely to the substance of the book, this will become an excuse for many to write the book off, and for others a piece of ammo to fire at ideological opponents.

In that way, and in many others, this does look quite a lot like the Reinhart-Rogoff contretemps, to which Mr Giles draws a parallel. The errors identified in their spreadsheet turned out to be far more embarrassing to the authors than a threat to their work. The attack on the Reinhart-Rogoff analysis showed that the authors had made a mistake in the figures that led them to identify a “discontinuity” in growth rates when public debt reaches a 90% of GDP threshold. But the analysis by economists at the University of Massachusetts, Amherst actually reinforced the finding that growth rates tended to be slower when debt levels were higher. The UMass work did nothing to undermine the broad dataset that rested beneath the 90% work (and beneath their book, and many other pieces of research). Subsequent analyses have turned up a discontinuity at various thresholds (though other papers have not). And at any rate the key debate, over causation, is mostly unresolved. But none of that mattered when the news broke. Partisans took to their bunkers to lob bombs at each other, the truth of the matter be damned.

It would be unfortunate were that to happen in this case, but it almost certainly will, and indeed it has already begun. But it’s worth looking at the bright side. In this case, the economist in question put his data and calculations online from the outset. That allowed an industrious journalist to take a closer look at the analysis. The truth will out for those who care to look for it, and the process will—hopefully—improve incentives for everyone involved. Academics will learn that putting data online is not enough; the work should be carefully documented if allegations of impropriety are to be avoided. And the FT will sell a lot of copies of today’s paper, demonstrating that careful data analysis can be good for business. And in time, it will be clear enough whether Mr Piketty is the prophet some reckon or something else.

– See more at: http://www.economist.com/blogs/freeexchange/2014/05/inequality-0?fsrc=scn/tw/te/bl/ed/pikettyproblem#sthash.oMC3Ej99.dpuf