No results.

How (and why) not to rank academics

GD Star Rating

The recently-launched Microsoft Academic Search, a product of Microsoft Research Asia, has made a bit of a splash as a potential competitor to Google Scholar. Although its coverage does not seem as detailed as Google Scholar quite yet, MS Academic Search has a number of additional features, such as author and conference pages, publication activity graphs, and the like. (It also has a really unwieldly, eight-syllable name; let me abbreviate to MSAS.)

Amongst its features, MSAS provides rankings of “Top Authors” within various fields of computer science. This is, needless to say, not something to be done lightly: academia is built around status and reputation, and to rank academics is to make a statement about their status, which a disclaimer buried in the explanatory notes does little to diminish. And in this case, MSAS should have thought bit more carefully before they decided to publish these rankings.

In MSAS, authors are ranked by the number of in-domain citations they receive within a given time period. This is a dubious enough metric in itself. But even bearing the questionable methodology in mind, MSAS comes up with some slightly surprising top-ten entries in their list of top information retrieval authors of all time , and some even more surprising ones in their ranking of top authors in the last five years.

There are, I suspect, a number of problems with how MSAS is calculating citation counts, but a quick browse through the cited-paper lists of some highly-ranked authors shows that their most egregious aberration within the field of information retrieval is that they are including TREC overview papers in their citation counts.

Now, TREC overview papers attract a lot of citations, because whenever anyone uses the test collection developed at a TREC task, they cite the corresponding overview paper. But TREC is not a peer-reviewed conference. And the overview papers in particular are not research publications at all; they are, rather, a summary of organizational information, participant descriptions, and result statistics. Track organizers put a lot of effort into the tracks, and deserve recognition for this effort in other ways; but counting citations to their overview papers is like ascribing conference citations to the person who wrote the preface to the proceedings.

To be fair, Google Scholar also includes TREC papers in their list of academic publications. But then Google Scholar does not attempt to rank academics. This may simply be laziness on Google’s part, but it might also be a recognition that this is tricky ground to be stepping on. Providing a slight specialization of a search interface is one thing; taking it upon yourself to summarize and rank the publication career of researchers is another. If you’re going to do the latter, you had better pay some attention to doing it correctly.

MSAS has some interesting and useful features. But, as it stands, the author ranking is not one of them. It should be taken down until they get their data and methodology right.


Leave a Reply




You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>