Test your robots.txt file

  1. Open the tester tool for your site, and scroll through the robots.txt code to locate the highlighted syntax warnings and logic errors. The number of syntax warnings and logic errors is shown immediately below the editor.
  2. Type in the URL of a page on your site in the text box at the bottom of the page.
  3. Select the user-agent you want to simulate in the dropdown list to the right of the text box.
  4. Click the TEST button to test access.
  5. Check to see if TEST button now reads ACCEPTED or BLOCKED to find out if the URL you entered is blocked from Google web crawlers.
  6. Edit the file on the page and retest as necessary. Note that changes made in the page are not saved to your site! See the next step.
  7. Copy your changes to your robots.txt file on your site. This tool does not make changes to the actual file on your site, it only tests against the copy hosted in the tool.

Testing Page: https://www.google.com/webmasters/tools/robots-testing-tool

Source Link: https://support.google.com/webmasters/answer/6062598?hl=en

Asked question