Keyword searching is so Web 1.0. (Now its about A.I.)

Posted by Jim Stroud

August 25, 2006

Hmm… If Barney Pell has his way, keyword searching for resumes may be a thing of the past. I’m watching this guy. Check out a snippet from an article I just read…

..snip-snip…

In the eyes of a search engine, the Web is essentially a body of words on billions of pages, along with the hyperlinks that connect the words. One of Google’s big breakthroughs was to link those words efficiently, measuring relevance by the appearance of words on a page, and the number of hyperlinks pointing to that page, or its popularity.

As a rule, search engines don’t understand the words–they’re merely programmed to match keywords that are more significant on a page, closer together or linked more often from other pages. So in essence, when someone types in “woolly mammoth,” he or she sends the search engine on a wild goose chase for those words, not the animal.

As a result, search engines miss the nuance of the human language. For example, Google might render a simple search for “books by children” by scouting for the pages that include the words “books” and “children,” but it would eliminate the so-called stop word–in this case, “by”–because stop words are those that occur on almost every page. Yet those stop words occur so often because they are important to the meaning of a phrase. “Books by children” is different from “books about children,” and different still from “children’s books.”

Barney Pell, founder of a yet-to-be-launched AI search engine, calls the restrictive language of search engines “keywordese.”

“Search engines try to train us to become good keyword searchers. We dumb down our intelligence so it will be natural for the computer,” said Pell, whose company, Powerset, is based in Palo Alto, Calif.

“The big shift that will happen in society is that instead of moving human expressions and interactions into what’s easy for the computer, we’ll move computers’ abilities to handle expressions that are natural for the human,” he said.

Powerset, which hasn’t divulged its launch date yet, is using AI to train computers not just to read words on the page, but make connections between those words and make inferences in the language. That way a search engine could think through and redefine relevance beyond the most popular page or the site with the most occurrences of keywords entered in a search box.

READ: Spying an intelligent search engine