ABOUT SEARCH ENGINES

Discuss Computer Science and Programming related problems

Moderators: bristy1588, Labib

MATHPRITOM
Posts: 190
Joined: Sat Apr 23, 2011 8:55 am
Location: Khulna

ABOUT SEARCH ENGINES

Unread post by MATHPRITOM » Thu Dec 29, 2011 11:53 pm

HOW SEARCH ENGINES WORK IN INTERNET?

User avatar
*Mahi*
Posts: 1175
Joined: Wed Dec 29, 2010 12:46 pm
Location: 23.786228,90.354974
Contact:

Re: ABOUT SEARCH ENGINES

Unread post by *Mahi* » Fri Dec 30, 2011 12:01 am

Please read Forum Guide and Rules before you post.

Use $L^AT_EX$, It makes our work a lot easier!

Nur Muhammad Shafiullah | Mahi

qeemat
Posts: 11
Joined: Thu Feb 16, 2012 6:43 pm

Re: ABOUT SEARCH ENGINES

Unread post by qeemat » Fri Feb 17, 2012 4:41 pm

It has been estimated that the amount of textual information accessible via search engines is at least 40 times larger than the digitized content of all the books in the Library of Congress, the world's largest library. It is a challenge to provide access to such a large volume of information, yet current search engines do remarkably well in sifting through the content and identifying related links to queries.

There is a multitude of information providers on the web. These include the commonly known and publicly available sources such as Google, InfoSeek, NorthernLight and AltaVista, to name a few. A second group of sources--sometimes referred to as the "hidden web"--is much larger than the public web in terms of the amount of information they provide. This latter group includes sources such as Lexis-Nexis, Dialog, Ingenta and LoC. They remain hidden for various reasons: they may not allow other information providers access to their content; they may require subscription; or they may demand payment for access. This article is concerned with the former group, the publicly available web search services, collectively referred to here as search engines.

Search engines employ various techniques to speed up searches. Some of the common techniques are briefly described below.

Preprocessed Data

One way search engines save time is by preprocessing the content of the web. That is, when a user issues a query, it is not sent to millions of web sites. Instead, the matching takes place against preprocessed data stored in one site. The preprocessing is carried out with the aid of a software program called a crawler. The crawler is sent out periodically by the database maintainers to collect web pages. A specialized computer program parses the retrieved pages to extract words. These words are then stored along with the links to the corresponding pages in an index file. Users' queries are matched against this index file, not against other web sites.

Smart Representation

In this technique, the representation for the index is carefully selected with an eye toward minimizing search time. Information scientists have produced an efficient data structure called a tree that can guarantee significantly shorter overall search time compared with searches conducted against a sequential list (see sidebar). To accommodate searches conducted by many users simultaneously and eliminate "wait queues," the index is usually duplicated on multiple computers in the search site.

Prioritizing Results

The URLs or links produced as a result of searches are usually numerous. But due to ambiguities of language (for instance, "window blind" versus "blind ambition"), the resulting links would generally not be equally relevant to a user�s query. To provide quicker access to the most relevant records (and to place them at or near the top), the search algorithm applies various ranking strategies. A common ranking method known as term-frequency-inverse document-frequency (TFIDF) considers the distribution of words and their frequencies and generates numerical weights for words signifying their importance in individual documents. It produces word weights whereby words that are highly frequent (such as 'or,' 'to' or 'with') and that appear in many documents have substantially less weight than words that are semantically more relevant and appear in relatively few documents.

In addition to term weighting, web pages can be weighted using other strategies. For example, link analysis considers the nature of each page in terms of its association with other pages�namely if it is an authority (number of other pages that point to it) or a hub (number of pages it points to). The highly successful Google search engine uses link-analysis to improve the ranking of its search results.

Hasib
Posts: 238
Joined: Fri Dec 10, 2010 11:29 am
Location: খুলনা, বাংলাদেশ
Contact:

Re: About Search Engines

Unread post by Hasib » Sat Mar 10, 2012 12:40 am

For learning this, you have to join GOOGLE! lol
A man is not finished when he's defeated, he's finished when he quits.

shayanjameel08
Posts: 10
Joined: Mon Nov 04, 2013 6:17 pm

Re: ABOUT SEARCH ENGINES

Unread post by shayanjameel08 » Sat Nov 09, 2013 11:00 am

The good news about the Internet and its most visible component, the World Wide Web, is that there are hundreds of millions of pages available, waiting to present information on an amazing variety of topics.

Post Reply