The Hundredth Monkey Phenomenon

In a recent post entitled Enterprise Search Crisis, Dave Kellogg, the CEO of MarkLogic, and XML database company, references the worst article on the state of Enterprise Search I’ve ever seen, and claims two points:

• The rock is database management systems. Many search solutions are integrations of relational databases with search engines along with templates for specific applications. While many search vendors are trying to reposition their products as application platforms, they’re not. Tying together MySQL, search engine X, and some pre-processing logic so you can properly feed the search engine indexer is not a great “platform” on which to build applications. Databases are much better application platforms and the real problem has been that databases, until recently, didn’t do content. But as new generations of database management systems — like MarkLogic — emerge, it will become increasingly clear that the platform for content applications should not be an enterprise search engine (bolted to other things), but instead a database management system built to natively handle content.
• The hard place is the Google Appliance. There will always be a need for “Google inside your company” type search. I call this the “crawl and index” value proposition. Given cost and complexity, I can’t see why Google won’t sweep up most of the market here. (I just wish they could do better with PDFs and email.)

I wholly agree with his first point, and don’t agree with his second; I don’t think that Google’s Search Appliance is the “hard place” here. There’s room to improve in just about every type of software imaginable, including enterprise search, but I really don’t think Google’s the right company to do it. I think the premise of the argument here is that, since search is becoming something of a commodity, Google’s plug-and-play search appliance is going to be hard to beat, but I honestly don’t believe they’ve got a great enterprise offering, from both pricing and capabilities perspectives. Don’t get me wrong, I really like Google, and they’ve built software that I rely on all the time, but it just doesn’t look like, to me, Google has really made a concerted effort to attack the enterprise market. I have a theory that their play with their appliance is to leverage their corporate name to drive sales rather than superior technology (deliberately; this is not a poke at their engineers who are among the best in the business).

Here are some supporting reasons:

1. the page rank algorithm doesn’t work on data that is very sparsely linked, and enterprise data is not very well linked
2. the Google Search Appliance’s pricing scheme works very well for small businesses but gets relatively expensive very quickly as you add more nodes for capacity. This means that, for any volume of data, they’re competing with players that have much better enterprise solutions, such as Endeca, FAST Search and Transfer, and Autonomy (disclaimer: I’m an Endeca employee, but this opinion is my own and does not reflect the opinion of Endeca)
3. search of the “I just want a search box” kind can be done in a commodity way with Lucene + a free crawler. This approach is fine for many small companies who just want a search box since their data is usually not in a 100 servers requiring a complex crawl. Based on point 2 above, Google’s pricing model means that they’re competing primarily with Lucene on the low end of the enterprise search market
4. their appliance is not tunable and not flexible; many enterprise IT shops demand flexibility and the ability to customize search behavior, the ranking algorithm, and basically anything you can imagine (not all at once, but in aggregate between all potential customers and all potential custom feature demands)

However, the biggest reason that Google’s Appliance is not going to rock the enterprise is that enterprise search is not just about search. The enterprise problem is that there is a ton of data and employees need to leverage it to make decisions. However, they are completely overwhelmed by the amount of data out there and therefore must have access to pertinent data subsets or summaries. Search can be good for finding specific things that you know exist a priori in that amount of data (needle in a haystack problem). It is not good at solving the harder problem of leveraging huge amounts of data for every day decision making. No, I’m not talking about traditional BI here. Guided Navigation, Relational Navigation, web analytics, BI, etc., are all other parts of the enterprise information access problem that Google’s search appliance doesn’t begin to address. Questions like: “who can I talk to about the Big Pharma?” “What customers in the Northeast are due for an upgrade in the next quarter?” “Where’s the best place in my supply chain where I can get a 4mm bolt that can withstand a ton of heat?” Answering those everyday questions is hard, and should be at the core of any enterprise information access solution.

So, yes, search itself is becoming a bit of a commodity on the lower end (competing with Lucene & custom solutions). However, search is by no means the end goal for information access companies, and Google has yet to show much interest in going beyond the search box.