(1)
Back in July we reported that Bing launched, or re-launched, its robots.txt tester tool. But on Friday, September 4th, Bing posted the official announcement on its blog. I am not sure what is new since when we posted outside of the download button but maybe I am missing something. But even that looks the same to me, so I guess this is just the official announcement?
Bing wrote "Webmasters can submit a URL to the robots.txt Tester tool and it operates as Bingbot and BingAdsBot would, to check the robots.txt file and verifies if the URL has been allowed or blocked accordingly. Not only this, but the test functionality checks the URL which we have submitted against the content of the editor and hence, once changes are made in the editor, you can easily and instantly retest the URL to check for errors. The system checks for allow/disallow statements for respective user agents and displays the robots.txt file in the editor with 4 variations i.e. http://, https://, http://www, https://www. Webmaster can edit the txt file and/or download the same to be updated offline. If there have been changes to the robots file elsewhere and updated, the webmaster can use the Fetch latest option to get the latest robots file of the property."
When you click "proceed" on the tester tool:
You get this overlay box letting you:
(1) Download the updated robots.txt file. Save the updated robots.txt file from the editor.
(2) Upload the file to your domains root. Upload the updated robots.txt to your domains root, then check your updated file is the latest version.
(3) Request Bing to update. Submit a request to let Bing know your robots.txt file has been updated.
The download option provides a step-by-step process of updating the file that includes downloading the edited file, uploading the same on the domains root which can be checked for a live version and lastly requesting Bing to update the same.
Fabrice Canal from Microsoft said "As robots.txt rules can be extremely challenging to understand and edit especially when using wildcard character *. By offering this easy to use robots.txt testing tool, we offer transparency on how we process robots.txt and help webmasters to edit and verify their robots.txt."
Forum discussion at Twitter.
