While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. Recently, I realized that the website was not showing up on google search results, upon further investigation, i realized that the old robots.txt file was still there (even though I can't locate the file in my root folder).

.txt

I inserted another robots.txt file in my root folder, this time with the allow option instead of disallow. But it doesn't update and still blocks google from accessing the site.

While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. Recently, I realized that the website was not showing up on google search results, upon further investigation, i realized that the old robots.txt file was still there (even though I can't locate the file in my root folder).

http://youngauthors.my/robots.txt

I inserted another robots.txt file in my root folder, this time with the allow option instead of disallow. But it doesn't update and still blocks google from accessing the site.

• google
• robots.txt

Share Improve this question asked Nov 13, 2017 at 7:49 Gan Zhe JingGan Zhe Jing 322 bronze badges 1

• 1 Cross-site duplicate – unor Commented Nov 15, 2017 at 2:52

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

Just delete file robots.txt. WordPress is generating virtual robots.txt file, and Yoast SEO plugin (and other plugins too) allows you to customize it. But, if you have an actual robots.txt file, it will prevent virtual one to work.