No results.

Detailed guidelines don’t help assessors

Social scientists are often accused of running studies that confirm the obvious, such as that people are happier on the weekends, or that having many meetings at work make employees feel fatigued. The best response is, what seems obvious may not actually be true. That, indeed, is what we found in a recent experiment. We [...] → Read More: Detailed guidelines don’t help assessors

“Approximate Recall Confidence Intervals”, updated and in submission

Much later than I intended, after painstaking editing to get the length down from 39 to 31 pages, I’ve prepared a revised version of “Approximate Recall Confidence Intervals”, which is now in submission. Aside from tightening up the text and excluding a few inessential results, the main change from the first version has been to [...] → Read More: “Approximate Recall Confidence Intervals”, updated and in submission

Recall confidence intervals

Frequent readers of this blog will know of my burning desire to move IR research away from dry technical topics and towards questions that directly impact and excite the retrieval user. In pursuit of this goal, I have for the past year been working on a paper on estimating two-tailed confidence intervals for recall under [...] → Read More: Recall confidence intervals

Attention-enhancing information retrieval

Last week I was at SWIRL, the occasional talkshop on the future of information retrieval. To me the most important of the presentations was Diane Kelly’s “Rage against the Machine Learning”, in which she observed the way information retrieval currently works has changed the way people think. In particular, she proposed that the combination of [...] → Read More: Attention-enhancing information retrieval

How accurate can manual review be?

One of the chief pleasures for me of this year’s SIGIR in Beijing was attending the SIGIR 2011 Information Retrieval for E-Discovery Workshop (SIRE 2011). The smaller and more selective the workshop, it often seems, the more focused and interesting the discussion. My own contribution was “Re-examining the Effectiveness of Manual Review”. The paper was [...] → Read More: How accurate can manual review be?

Assessor disagreement and court sanctions

I mentioned Cross and Kerksiek’s suggestion of vocabulary discovery in my previous post. Their paper also contains an interesting reference to a case (Felman Products, Inc. v. Industrial Risk Insurers) in which the defendant was penalized for the carelessness of their production. The defendant inadvertently produced privileged documents, and sought to have them [...] → Read More: Assessor disagreement and court sanctions

Corpus characterization in e-discovery

In e-discovery (document retrieval for civil litigation), one side has the documents, the other side proposes the query. This creates an information asymmetry; the requesting side cannot view the corpus to decide what keywords to use and what queries to propose, and opportunities for query iteration are limited, expensive, and liable to being contested.

What [...] → Read More: Corpus characterization in e-discovery


Harvard researcher and open-access advocate, Aaron Swartz, faces 35 years’ jail for downloading 4.8 million articles from JSTOR. Still relaxed and comfortable about publishing in closed-access journals?
Correct spelling and grammar more important than positivity or negativity of product reviews — Panos Ipeirotis.
Fitting an elephant with four parameters.
Placebos as effective as real medicine in improving [...] → Read More: Outlinks

Multiple significance tests in IR

At the most recent TIGER reading group, Mark Sanderson presented Bland and Altman’s introduction to multiple significance tests and the Bonferroni method. The basic point is simple: if you keep trying different experiments, and testing each for significance, then eventually you will find significance by chance, even where no real effect exists. Therefore, [...] → Read More: Multiple significance tests in IR

PhD Thesis

My PhD thesis was passed (actually a few months ago), and I’ve placed it online. The core of the research material has been published elsewhere, but there are a few updates:

Score standardization (Chapter 4): The chapter on standardization combines Score Standardization for Robust Comparison of Retrieval Systems (ADCS 2007) and Score Standardization for [...] → Read More: PhD Thesis