Robots.txt is actually a file that exists on the root directory of each Site and can be employed to instruct search engines like yahoo on which directories/data files of the web site they're able to crawl and involve within their index. In case you are expecting them to be a https://news.rafeeg.ae