Robots.txt is usually a file that exists on the foundation Listing of every Internet site and can be used to instruct search engines like google and yahoo on which directories/documents of the web site they could crawl and consist of in their index. be to make use of this ingredient https://news.rafeeg.ae