[fabulouslyfunkyfinds123]

Hi,
Ans is= A crawler is also referred to as a robot or bot from a search engine. A crawler reads the content from a page and usually stores this in a search engines database, called an index. From this index the search results are extracted when one has entered a keyword in a search engine.
There are also other crawlers, like the Wayback-machines crawler, that just stores the content(also images)


Thanks and regards
Partap

[myroncarlin]

Make your site search engine friendly.

[manohar]

I think RSS submition is best options crawl on your site .............

[vishal9717]

Hi there

The Spider (crawler) at Work

They are called spiders because they crawl over the Web in search of content. Search engines gather data about a website by ‘sending' the spider or bots to ‘read' the site and copy its content. This content is stored in the search engine's database. As they copy and absorb content from one document, they create record links and send other bots to make copies of content on those linked documents this process goes on and on. As of now the major search engines have established databases that measure their size in the tens of billions using this process.

Spiders are designed to read site content like a human. It will start at the top left hand corner, reading line by line. In case of columns spiders will read the left hand column before moving to central and right hand column. It will read and record the whole site to the very last word.

[w.cain]

Thanks for sharing information regarding search engines crawlers or spiders.
That is true. All the information are correct. This will help other members on this forum. Thank you.

[manohar]

A crawler is also referred to as a robot or bot from a search engine. A crawler reads the content from a page and usually stores this in a search engines database, called an index. From this index the search results are extracted when one has entered a keyword in a search engine.

[skill-games]

A crawler is also referred to as a robot or bot from a search engine. A crawler reads the content from a page and usually stores this in a search engines database, called an index. From this index the search results are extracted when one has entered a keyword in a search engine.
There are also other crawlers, like the Wayback-machines crawler, that just stores the content(also images) from a website to create a record of the internet.*