The Hundredth Monkey Phenomenon

In one of the best posts on search I’ve read in a long time, Dave Kellogg observes “Why guess when you can know?”. That is, a great deal of innovative energy in the search industry in general is going into improving keyword search (using semantic knowledge, statistical analysis, etc.) or into natural language search (e.g. Ask.com, some BI tools). He follows the post with another post that summarizes his main point:

Here’s a quick follow-up to the last post, which got long and perhaps failed to net-out the point as clearly as it might have. Here’s the point:

• Search engines seem to assume that the question is improving relevancy based on a few keyword grunts.
• They use various degrees of magic to try and improve relevancy: dynamic clustering, taxonomy, recent query history, social tagging/editing, entity extraction, PageRank, SemanticRank, SomethingRank, etc.
• The point of all this magic is to guess exactly what you want.

Here’s the question: why guess when you can know? When you send a SQL query to a Oracle, it’s not guessing what you want (e.g., show me average sales by product line for 2Q). It knows what you want and there is a single correct answer to your question.

Yes! That’s exactly it! Take advantage of the structure to build applications that allow naive users to unknowingly ask complex questions/queries. A user shouldn’t have to know SQL (or XQuery, or SPARQL, or whatever) to ask the questions. And, more importantly, the IT departments shouldn’t have to know beforehand what questions the users will ask. Let the users form the questions to ask the system as they need to. As I see it, that’s one of the issues with traditional BI; it requires people to maintain the types of reports that are generated.