A robots.txt file is a simple text file that instructs World wide web crawlers about which aspects of an internet site are open for indexing and which really should keep on being off-boundaries. It provides a set of rules, usually penned in an easy structure, that immediate crawlers like Googlebot https://www.seoclerk.com/user/n1affiliate