What is Robots.txt?

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

The Robots.txt file of a website will work when it is used as a request to specific robots to ignore directories or files specified within the Robots.txt file. Websites with sub-domains generally need a Robots.txt file for each sub-domain, all so that information that is not viewable to the public is not picked for a keyword search. It also heightens the keyword density of the actual web page text, and keeps visitors from coming across misleading or irrelevant to the keyword searches. Robot.txt protocols are simply advisory though. There is no law requiring websites to have Robot.txt files, or to use them on their web pages.

Call Brick Marketing today at 877-295-0620 or contact us and find out how we can help you drive visitors and sales for your business.