Predicting Highly Cited Scientific Papers

Lots of people make predictions. But very few—especially in the pundit world—are held accountable, or even reexamine their predictions. Recently, Mark Newman, a physicist and network scientist at the University of Michigan, decided to actually check his predictions.

Five years ago, Newman created a method of determining which scientific papers would be expected to be highly cited. This method was based on the "first-mover advantage". As he notes in the abstract:

Mathematical models of the scientific citation process predict a strong "first-mover" effect under which the first papers in a field will, essentially regardless of content, receive citations at a rate enormously higher than papers published later. Moreover papers are expected to retain this advantage in perpetuity -- they should receive more citations indefinitely, no matter how many other papers are published after them. We test this conjecture against data from a selection of fields and in several cases find a first-mover effect of a magnitude similar to that predicted by the theory. Were we wearing our cynical hat today, we might say that the scientist who wants to become famous is better off -- by a wide margin -- writing a modest paper in next year's hottest field than an outstanding paper in this year's.

Newman predicted at the time which papers would be successful. And in a preprint over at the arXiv, he reexamined those predictions to see how he did. And he nailed it!

Among the over 2000 papers in our original data set, we examine the fifty that, by the measures of our previous study, were predicted to do best and we find that they have indeed received substantially more citations in the intervening years than other papers, even after controlling for the number of prior citations. On average these top fifty papers have received 23 times as many citations in the last five years as the average paper in the data set as a whole, and 15 times as many as the average paper in a randomly drawn control group that started out with the same number of citations.

The full paper is well worth checking out, as Newman discusses controlling for factors such as cumulative advantage as well as how the papers he predicts to be successful going forward are predominantly from the top tier journals of Science and Nature, and what this means.

Top image:public domain