Secret war: Public search VS. Commercial search(2)

Future of Web search: Public search VS. Commercial search

As mentioned above, with the rapid increase of web pages, some bottleneck problems in coverage and update interval of current search engine have become very obvious. These make some experts rethink the basic architecture of current search engine. In this condition, the research for the public search engine reloaded. What¡¯s the future of search engine, public or commercial?

Now many peoples believe search engine administrates what you could know and what you couldn't reach to a large extent. If all the information is in the charge of a small group, the combination of this absolutely administration and commercial profit may be very dangerous to the benefit of common users. We want to know why so many search engine companies always discuss how they can make more money from advertisements but not propose a perfect solution for us. Just for this reason, it may be better to convert the web search engine in to a public service.

Moreover, almost all the Internet technologies, from TCP/IP to E-mail, are public opening technologies, but the better commercial service could be designed based on them. It may be a basic principle for the continued development of Internet. Search engine may also comply with this principle. We can find a better commercial mode for search engine. First, there should be a basic public opening web search system. Then, some companies can design better search system based on this public platform.
But we have to solve two problems. First, designing a public search system better than current commercial system. At least, this system should provide the same quality of search service as Google. Second, who are willing to build such system? Many excellent technologies failed just for this reason.
Some tips could be found in digital library research. We all knew Google came from Stanford digital library project (http://www-diglib.stanford.edu/). It's just a small part of this project. Only few experts knew the main research of this project, InfoBus system. The main goal of this system is to integrate different kinds of digital resources on Internet, which is a basic research topic of digital library. For some reasons, this system is not widely implemented. But after 20th century, many advanced technologies such as Webservice, GRID appeared. This digital library research topic could go ahead based on these new technologies.

HUST digital project is just for it (http://dris.hust.edu.cn/English/main.htm). This research group proposed DRIS (Domain resource integration system), whose goal is also to integrate all kinds of information resources and build an Internet information retrieval infrastructure. And very interested, here is also a small component for a web search, which is called web search engine based on DNS. In theory, this system can cover all the pages on Internet. Its update interval could even be one day. There have been some discussions for it in IETF. Its test bed is being built on CERNET (China education and research network, including all the universities in China).

Here is a very brief introduction for this system. WWW is a large distributed dynamic world, but most practical and commercially operated Internet search engines are based on a centralized architecture. So with the exploding of Internet information, this contradiction became very serious. This is just the origin of bottleneck problems of current search engine. So a better solution must apply a completely different architecture. Some research like SIREN in IRTF (research department of IETF) had some considerations for extending the DNS's navigation function to web pages search function. The hierarchical distributed architecture of DNS is an efficient architecture to manage the WWW. Not only for titles of web site, but also for their content. HUST Digital Library projects gives a practical solution based on this basic idea. A distributed system must be a public system. Moreover, its layered structure can also ensure it's a practical system. Its bottom layer provides the search engines in local networks of different organizations like universities. This is the basic impetus to build such system. Then these small scale search engines are integrated by other technologies to build a search engine covering entire WWW. More information could be found in its site.
This research may give a promising solution for a better public search engine. Although academic research for this topic is still in low tide, a public search system may be inevitable.

Some principles are available at anytime. First, in any free market, customer should be God forever. Second, the evolutionism in technology, only the technology that meet the demand of common users better could survival.


Author
Wang Liang
Currently, Ph.D in informatics school at Huazhong university of science and technology (HUST).One of main directors of HUST digital library project

Two similar reports:
http://www.circleid.com/article/588_0_1_0_C
http://www.trnmag.com/Stories/2004/0...ch_040704.html