Why Has Search Become So Hard?

June 12, 2006 

By Francois Bourdoncle

One decade ago, the average PC hard drive offered 100MB of storage capacity, according to the Computer History Museum in San Jose, Calif. Today, the average hard drive in a home PC holds 2,500 times more storage, a necessity given the incredibly large amounts of data we generate each and every day.

Whether it’s in the form of e-mail, text documents or multimedia files, having the right information at our fingertips is essential for making business decisions. We rely on information to learn about our companies, our customers, a given market and key competitors. We make purchasing decisions based on information. And more and more, we turn to the Web for information, often before searching our own PCs or company databases. That’s because search inside a business has become too difficult.

We all possess the ability to search smarter and to search easier. You wouldn’t buy a sleek, shiny new sports car if it didn’t have a powerful engine, would you? The same thinking can be applied to choosing search software.

Here are seven tips to help find the right search engine for your business and search better in the process.

1. Search Should Begin with Your Desktop
Whether you’re searching for an e-mail, a file on the server or a document on the Web, all searches begin from your desktop. In recent years, the growth of unstructured data has made the corporate desktop the key portal to access data within the organization. With this heightened awareness, dozens of vendors have released products for both business and consumer use. To help evaluate and determine which desktop search solution is right for you and your organization, consider the following questions:

• Does the solution respect user privacy? Does it respect the existing security policies within your organization?
• Is it easy to use for both novice and power users? Is there a range of advanced search features available?
• Will your IT administrator be able to deploy the software and manage updates from a single console, or set access policies?
• Can the solution index the content within your organization? Does it offer support for a variety of e-mail programs?

3. Understand the Two Schools of Search
Web search is different from enterprise search. Web or Internet search is the process of searching for content indexed on the World Wide Web, while enterprise search refers to accessing information within an organization—everything from the desktop to intranet websites, directories and databases. Employees often overlook the amount of information available to them because there isn’t an effective, easy-to-use search tool available. This often leads employees to turn to the Web for information, and then they end up re-creating content that may already have been available internally.

Of course, the answers don’t always lie within the organization; Web search plays a distinct role in business and how we work. Over the years, enterprise search has gained a reputation for being complex and expensive, while consumer search engines are free and easy to use, but “too simplistic” for business use. The reality is that many CIOs end up spending anywhere from two to 10 times the cost of the software in professional services to make the solution useful. That’s because some search solutions are application platforms rather than actual products, which makes integration a challenge. Additionally, taxonomy projects, which are mandatory for some scenarios, are incredibly costly and time consuming, adding months to your deployment schedule and negatively impacting your ROI. The search “sweet spot” is the middle ground between these two schools of thought.

4. You Don’t Have to Be a Librarian to Use Advanced Search
Boolean, for many of you, conjures up images of a small flavor cube. But in the information world, Boolean refers to a search method named for George Boole, a British mathematician. The idea behind Boolean search is that by adding a few simple words (AND, OR, NOT) to your query, you can generate more relevant results. These “operators” help find information based on the relationship between content (e.g., cats NOT kittens; cats AND kittens). There are many other advanced search options available today to help pinpoint the right information.

Today’s enterprise and Web search engines offer a basic range of advanced features, while others offer unique search capabilities. Imagine searching for an e-mail from a client in Tokyo without knowing how to spell his name. You can search phonetically (“sounds like”) and receive highly accurate results, or you can try a proximity search to return more relevant results. For example, you can search for a document that contains Volvo within five words of safe. You may not think you’ll ever need these features, but once you learn how to use them, you won’t look back.

5. A Search Engine Should Respect Your Security and Privacy
Many of you have probably downloaded a free desktop search product (or two) on your PC. But did you know that you may be putting yourself and your organization at risk? Some desktop search tools pose serious security threats. For example, some programs keep cached versions of older documents or recently visited files that have been deleted. Some desktop search tools may connect to the Internet without your knowing it, leaving your data exposed. Further, consumer desktop search tools don’t allow administrators to set fine-grained security policies to assign user privileges and control what content is indexed. When evaluating an enterprise search product for your organization, choose a solution that gives you the greatest control. Make sure the technology can be centrally installed, managed and customized, and doesn’t put your employees at risk.

6. Integration Is the Key to Easier Searching
While the current market offers many different products designed to address desktop and enterprise search, consider a solution that will offer users a single point of access to information, whether it resides on one’s PC, the company intranet or a data silo on the other side of the globe. Forcing users to learn separate tools to search different data repositories will only have a negative effect on user adoption and impact your overall ROI. By selecting a unified search technology that covers desktop, enterprise and Web search, you will keep users happy.

7. Think Globally
Whether you’re a small business or a large enterprise, you probably deal with business partners and customers around the globe. If you don’t yet, you will. Therefore, your search solution should also think globally and offer support for a variety of different languages—especially complex Asian and Middle Eastern languages as well as Dutch, an especially vexing language for search engines. This will also make the task of extending the search solution to additional subsidiaries a smooth and simple process.

End Notes
As long as we continue to produce data, search in all of its forms will continue to play an essential role in knowledge and content management. Over the next few years, the enterprise search market will undoubtedly show increasing signs of consolidation. On the consumer side of search, expect to see more and more verticalized Web search engines designed for searching specialized content.

Francois Bourdoncle is the president, CEO and co-founder of Exalead. Bourdoncle was involved in the development of the programming languages Jazz and Exascript. He was a researcher at both Digital Paris Research Laboratory and Digital Systems Research Center in Palo Alto, Calif., and holds a PhD in computer science from Paris-based Ecole Polytechnique.