When I try to google search my web address, the follow comes up as the meta description

"A description for this result is not available because of this site's robots.txt – learn more."

The website I have tried to edit the robots.txt (downloaded the WP Robot plugin) and although I have changed it to

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The robots.txt still appears as

User-agent: *
Disallow: /

I have Yoast SEO, WP Robots, and Salient's Theme Nectar. If anyone might be able to provide me with a solution that would update the robots (or where to go from here) that would be incredibly helpful!

When I try to google search my web address, the follow comes up as the meta description

"A description for this result is not available because of this site's robots.txt – learn more."

The website I have tried to edit the robots.txt (downloaded the WP Robot plugin) and although I have changed it to

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The robots.txt still appears as

User-agent: *
Disallow: /

I have Yoast SEO, WP Robots, and Salient's Theme Nectar. If anyone might be able to provide me with a solution that would update the robots (or where to go from here) that would be incredibly helpful!

• seo
• robots.txt

Share Improve this question edited Dec 9, 2015 at 9:20 Pieter Goosen 55.4k2323 gold badges115115 silver badges210210 bronze badges asked Dec 9, 2015 at 9:05 ConfusedConfused 8111 gold badge11 silver badge33 bronze badges 3

• 2 Have you tried to edit the robots.txt file directly (without using plugins)? Have to enabled the search engine visibility? Settings -> Reading -> Search Engine Visibility (Uncheck this option) -> Save – Nikhil Commented Dec 9, 2015 at 9:12
• 3 Have you tried clearing your cache? You have an X-Powered-By: W3 Total Cache/0.9.3 in your response header for the robots.txt file. – RRikesh Commented Dec 9, 2015 at 9:26
• 1 Are you using any plugins for caching or security – Pratik bhatt Commented Mar 29, 2017 at 4:34

Add a comment  | 

4 Answers 4

Sorted by: Reset to default 3

Once I experienced the same issue, this is what I did to fix the issue.

Edit the robots.txt file directly (using FTP/SSH),

User-agent: * 
Disallow: /wp-admin/ 
Disallow: /wp-includes/

There are two reasons if the robots files not updated when you edited using a plugin.

1. File permission.
2. Some other plugin is reverting the changes.

Also try to update the search engine visibility settings (this might not be an issue, but just have a look).

Settings -> Reading -> Search Engine Visibility (Uncheck this option) -> Save

I hope this helps.

I had the same issue and for me, I had to purge the Cloudflare cache.

A WordPress site doesn't need an actual robots.txt file. WordPress itself will intercept requests for robots.txt and respond with its own version, and that's what your plugin is most likely trying to modify.

Delete or rename your actual robots.txt file and try it.

Happened to me too, try to "Purge All Caches" on Wordpress settings/hosting service provider.

Then to check, open a browser in "Incognito mode" (chrome, make sure you didn't have any Incognito windows open already, otherwise you'll have to close them all and re-open a new Incognito window), then head over to your www.mydomain/robots.txt You should now be able to see the updated content file like expected.