Topic: What is robots.txt used for?

I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

Please guys clear it.

Re: What is robots.txt used for?

The robots exclusion standard, also known as the robots exclusion protocol or robots.txt protocol, is a standard used by websites to communicate with web crawlers and other web robots.

https://en.m.wikipedia.org/wiki/Robots_ … n_standard

Re: What is robots.txt used for?

robots.txt is a way for you to control how frequent your site will be crawled to avoid overload your site and not get banned by your host

Volunteer admin at 000webhost.com
http://OngETC.com - http://uHostETC.com

Re: What is robots.txt used for?

Here you go! A great place to get some info http://bfy.tw/4ft3. No hard feelings big_smile

We have moved the community http://hosters.cc. I'll still pop in on here to see how things are going.