(61)
Google is now sending out a rush of new warnings via Google Search Console, formerly Google Webmaster Tools to notify webmasters that GoogleBot cannot access their CSS and JS (JavaScript) files on their web sites.
Michael Gray first notified me directly via Twitter and I planned on covering it tomorrow, but it seems Google is sending these warnings out at a rapid pace. Tons of webmasters are concerned after receiving these warnings.
Here is a picture:
Google has been telling webmasters to not block CSS & JavaScript for years and years. Here is Matt Cutts in 2012 telling webmasters not to block it. The webmaster guidelines were updated to say not to block them. The new fetch and render tool warns you when you block CSS and JavaScript. We also know, Google renders the page as a user would see it these days, so blocking CSS/JS can impact that big time.
Like I said, it seems like Google is sending these notices out in mass quantity now. The message reads:
Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly so blocking access to these assets can result in suboptimal rankings.
This is not a penalty notification, but a warning that if Google cannot see your whole site, it may result in poorer rankings.
If you get this message, talk to your developers and discuss what you can do, if you need to do anything. Use the fetch and render tool to diagnosis the issue deeper as well.
Forum discussion at Twitter.
Update: I should add, that many many WordPress sites are getting this notification because their /wp-includes/ folder is blocked by robots.txt. Plus there are many popular CMS solutions that block their include files by default.
Update 2: I have more details from Google on this notification, which you can read over here.
Comments:
Malik
07/28/2015 03:14 pm
how to fix this warning ?
Andy McIlwain
07/28/2015 03:17 pm
I suspect WordPress-powered sites are being alerted en masse if their /wp-includes/ folder is blocked by robots.txt (which many are). The core jQuery included in WP is enqueued from /wp-includes/. Edit: The /wp-includes/ blockage was removed in 4.0 thanks to a patch from Joost de Valk (Yoast): https://core.trac.wordpress.org/ticket/28604 Another reason to update your WordPress install. :)
RyanMJones
07/28/2015 03:19 pm
Came here to say exactly this.
Barry Schwartz
07/28/2015 03:21 pm
yep, i guess i should add that to the post above.
Jacob Zucchi
07/28/2015 03:22 pm
Drupal and Magento too
Sunshine Papillons
07/28/2015 03:22 pm
I have this just pop up, should I go and delete wp-includes now from the robots? Or is it fine?
Andy McIlwain
07/28/2015 03:25 pm
I think you're fine to remove it. Joost de Valk (Yoast) recommended as much earlier this year: https://yoast.com/wordpress-robots-txt-example/
RyanMJones
07/28/2015 03:48 pm
There's no reason to block wp-includes. Just block wp-admin and you're set.
Paolo Bianchi
07/28/2015 03:49 pm
Joomla too
RyanMJones
07/28/2015 03:50 pm
Go to webmaster tools. Do a fetch as Google. Unblock the .css and .js that are on your site that it complains about. (it may show some 3rd party files, but you have no control over those so don't worry about them)
omurphy
07/28/2015 04:02 pm
Actually, all of my Wordpress sites received this warning because /wp-admin/ was being blocked by the robots.txt. This is actually done by default in Wordpress (https://core.trac.wordpress.org/browser/tags/4.2.2/src/wp-includes/functions.php#L0 = scroll down to the do_robots() function) and is not from any plugin. When I used the Fetch as Render tool in GWT and saw what Google has rendered, Google bot was being blocked from accessing some scripts under /wp-admin/. I think the admin-ajax was one of them. I solved this by filtering the do_robots() function to remove the robots.txt /wp-admin/ disallow entirely. I then re-ran the Fetch as Render tool and Google bot was no longer being blocked. Here's the code I added to my theme functions.php file if anyone is interested: add_filter( 'robots_txt', 'jwd_remove_wp_admin_block', 10, 2 ); function jwd_remove_wp_admin_block($output, $public) { if ( !$public ) { return $output; } else { $output = ""; return $output; } }
omurphy
07/28/2015 04:04 pm
In my case it was only the /wp-admin/ which was being blocked in the robots.txt across all my updated Wordpress sites. At any rate, at the suggestion of Yoast (see here: https://yoast.com/wordpress-robots-txt-example/#updates), I went ahead and removed that wp-admin block which seems to have fixed the issue.
Big Boy Travel
07/28/2015 04:25 pm
so why doesn't Yoast just do a quick update for all users with it removed?
Michael A. Gray
07/28/2015 04:30 pm
It doesn't show anything being blocked when you do the fetch. I tried 6 random pages, all were fine.
omurphy
07/28/2015 04:40 pm
Make sure you're doing 'Fetch and Render'
Mike Good
07/28/2015 04:53 pm
Hi Ryan, I got the email this morning. I've fetched and rendered. The only things listed as blocked are something related to Wordfence and 3 things related to ShareAholic. Does that mean I should be OK? Thanks!
Elena Gorsvan
07/28/2015 04:58 pm
When I do Fetch and Render it shows that Google sees my page without Adsense and Media.net ads while visitors see it with ads. I have no idea why Google doesn't see that. Otherwise there is no difference. Anyone knows what to do?
Jim
07/28/2015 05:37 pm
Ok, So how do I Unblock the .css and .js that are on your site?
President Trump
07/28/2015 05:42 pm
I asked my techs to block Google wherever they can. We get notices like this all the time of Google crying about something
Kia Kamgar
07/28/2015 06:10 pm
I too just got this email, but nothing in blocked in my robot.txt. Very odd. Have tested it, checked it and the file looks fine, I have even fetched as Google, and it returns no errors.
Sinan İŞLER
07/28/2015 06:16 pm
There is something stupid going on because we didnt block any js or css but we get notification...
Isabel Isidro
07/28/2015 06:40 pm
It is not only complaining about the wp-includes in our site, but also wp-admin, particularly wp-admin/admin-ajax.php. I don't mind unblocking the robots from wp-includes, but I really don't want robots accessing wp-admin. What are the downsides of unblocking wp-admin?
Jonathan Guy
07/28/2015 06:43 pm
We've just had this warning on pretty much all of the Wordpress sites on our server. Not at all impressed with this. If this is a Wordpress issue then Google and Wordpress really need to talk to each other. Whilst we may have the knowledge and resources to deal with this I doubt the majority of the world's Wordpress users have any clue...
Sinan İŞLER
07/28/2015 06:45 pm
There is no smoke if there is no fire. There is a big chance they will make ranking variable this mobile js+css block thingy.
RyanMJones
07/28/2015 06:46 pm
most ad networks block Googlebot from seeing the ads - this isn't an issue to worry about.
Elena Gorsvan
07/28/2015 06:50 pm
Thank you!
Alexis Avellan
07/28/2015 06:55 pm
I remember reading Yoast saying that one shouldn't need to block the WP files because WP wrote something in the code to ensure they wouldn't get indexed. So I wiped my robots.text clean only to see that Google HAD indeed indexed some of the behind the scene files. It makes my site appear messy in the search results. So if you do so you may run into that same issue.
Isabel Isidro
07/28/2015 07:05 pm
We had a massive attack 2 years ago, and one of the things we did was to put password protection to wp-login.php. That triggered an "increase in authorization errors" and even after removing the password protection, our traffic steadily decreased. It was brutal. So I am scared when receiving these error messages from Google and when they don't like something that we block on the site
Visualize Digital
07/28/2015 07:06 pm
I received this warning on all of my WP based sites and client's WP based sites that have WordFence (a WordPress security plugin) installed on them. In the search console log it only shows this instance: https://www.website.com/wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=E7EB73D7D15EA1319F69927D2D7BA03D&r=0.2138125274796039
Alexis Avellan
07/28/2015 07:13 pm
Likewise. Here Yoast says to not block those pages because Google won't like that but we don't want robots or individuals to be snooping around these pages. It's a mess.
shendison
07/28/2015 10:30 pm
@bigBoyTravel - Because Yoast doesn't / shouldn't / wouldn't / better not overwrite someones robots.txt file
shendison
07/28/2015 10:32 pm
Thanks Google for causing a cluster***k - In many of the emails today, examining the robots file showed nearly no restrictions - and yes, I do *not* want google indexing my wp-admin, for example, because it contains nothing for them.
Big Boy Travel
07/28/2015 10:51 pm
inherent in the Yoast plugin is a spot where they block WP_Admin & Plugin access to Google Bots. You go under SEO > Tools > Files to see it. It is part of their plug in and is part of the problem everyone is having today
shendison
07/28/2015 10:58 pm
If you mean SEO > Tools > File Editor and then look at robots.txt file, I really don't think that was generated by Yoast's plugin - At least it wasn't in our cases, where I looked at 11 sites today after this email, and they all had our own versions of robots.txt - (I know because we exclude /webalizer and /awstats and /cgi-bin too )
The Solution
07/28/2015 11:28 pm
Here is the solution for the issue if you are using WordPress. Only clearing the blocks from the robots.txt won't fix this every time. Check this tutorial, it is very simple, however your hosting should permit you to use and install without a problem a caching plugin like W3 Total Cache - http://learnthegeek.com/tutorials/eliminate-render-blocking-using-w3-total-cache.html
helene
07/29/2015 08:17 am
i don't see any msg error on my sites WP. what i should do ? also i don't know what we need do in this new case of googlebot ? Thanks for help!
Baron_Vendredi
07/29/2015 08:20 am
I can add that I build many sites with Moonfruit and yesterday had this message about only one of them. My only guess (and I am not as technically adept as many commenting here appear to be - hence I use Moonfruit) is that it is related to the way the mobile version of the site displays when you check it via Google - in that it "says" everything is well, but just displays a block of colour. Moonfruit tell me they are on the case, which is what I pay them for. Perhaps why Wordpress, Drupal and Joomla are also affected. Not an expert - just throwing in my observations for the public good.
Gary
07/29/2015 08:40 am
Build websites for users, not for Google...
John
07/29/2015 09:19 am
Hello, whats about external Scripts? Webmaster Tools shows me some external js-scripts which are blocked. Usually Webmaster Tools tries not to show scripts that you cannot control, but in some cases it seems that it doesn´t work. Any recommendation?
Jerry B
07/29/2015 10:08 am
Google also want us to stop blocking our arteries:) http://www.dailymail.co.uk/sciencetech/article-3177026/Google-wants-world-meat-free-Search-giant-tried-buy-veggie-burger-start-300-MILLION.html lol Google Food, another exciting move by Google into another industry
The Solution
07/29/2015 10:15 am
You can't resolve them. Facebook, Twitter, Google Fonts, yeah Google Fonts...all of them, you can't control them, so ignore these files.
The solutoin
07/29/2015 10:16 am
Google wants to render the site the way a user is seeing it. If you don't comply you will be outranked from a guy that comply, so it is against your own interest to avoid this issue.
The solutoin
07/29/2015 10:19 am
The issue can be viewed by using the PageSpeed Insights. I highly doubt that you don't have a blocked rendering scripts on top. More than 90% if not even more of the WP sites got blocking scripts above the fold. https://developers.google.com/speed/pagespeed/insights/ and run a test to your site than claim if you don't have an issue. If you don't comply you will be outranked from a guy that resolved these issues. So you have a free choice - fix it and compete among the others or don't fix it and get outranked by your competitors that fix the issue. It is really your own choice, if you want to go against your own interest, it is still your choice.
John
07/29/2015 10:26 am
thanks! the same is by trying to optimize the Google page speed. But we have to live with google :D
Michael J Stewart
07/29/2015 10:50 am
I checked out a friends website because he received this email warning last night. I checked the robots.txt using the Webmaster Tools robots.txt tester and although a physical robots.txt files didn't exist WMT shows the default virtual robots.txt that Wordpress has built in. I immediately created a robots.txt and uploaded it and then checked it with it WMT which reported no problems. Fetch as Googlebot fetched the page without reporting any problems. This morning Fetch as Googlebot just reports 'Error'. I'd imagine that if this does impact on site visibilty rankings are going to be dropping on mass. Most small businesses don't even have WMT setup so won't even be getting this email alerts.
Schul
07/29/2015 11:26 am
I have the same question. i checked my blog on https://developers.google.com/... and i found that i have between others errors this one: " Your page contains scripting resources 5 and 4 CSS resources that block the display of your page, and thus delay." But on webmaster tool i didn't fidn nay message! So ?
Ben Guest
07/29/2015 12:05 pm
What's the best practice here for Wordpress? I've seen people all over the place. We have A: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ User-agent: Googlebot Allow: *.css Allow: *.js OR B: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ User-agent: Googlebot Allow: /wp-admin/ OR C: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Allow: *.css Allow: *.js OR D: User-agent: * Disallow: /wp-includes/ The last thing you want is to allow /wp-admin/ as hackers can sniff that out so I'm discouraged by what people are suggesting as a solution out there by just removing /wp-admin/. B is my choice, however, would like other insight. Thoughts? Thanks!
Warren O'Donoghue
07/29/2015 12:17 pm
So now Google now wants to index wp-includes, wp-admin AND wp-login.php! So what happens if you've changed wp-login.php to something else, to maybe stop spammers and dipsticks or does Google think they've all gone away now or something, la la Wizard of Oz, Cloud Cuckoo Land... They'll be asking for our Username and Passwords next!!!
Ben Guest
07/29/2015 02:06 pm
Oops, I'm sorry. That should be: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ User-agent: Googlebot Allow: /wp-includes/ Tested and nothing is preventing the rendering of the page for Googlebot. Yay!
The solutoin
07/29/2015 03:32 pm
No need to allow* the bot, it is pointless. You shouldn't disallow wp-admin and wp-includes. Here is an example of a good robots.txt - A deleted robots.txt. You don't need it anyway. or User-agent: * Disallow: Sitemap: http://yourdomain/sitemappath.xml You DON'T need anything to allow/disallow more in a regular situation. Why would you want to specify this, just let Googlebot crawl and despite for itself. If you see some URL that shouldn't be in the SERPs, just use Google Search Console and remove it. SO simple. Why everyone wants to make his life harder by forcing the Googlebot to do or not to do something. Why are people overcomplicating things. You don't need robots.txt anyway! Ask John!
The solutoin
07/29/2015 03:34 pm
Google is listing its own scripts too. You don't have control over them, as well as you can't compress images from Twitter and Facebook...
The solution
07/29/2015 03:36 pm
You will find this by using Fetch as Googlebot / Fetch and Render and choose the smartphone and desktop versions. Then click on the result and you will see the blocked resources. Search Console can also say that your site was rendered "partially", because of this. You don't have to receive a message for this, but you have to fix it, now more than ever.
Michael J Stewart
07/29/2015 04:15 pm
It's true that you don't need a robots.txt, but if you use Wordpress and omit to create one Wordpress creates a virtual one that Google can read.
Big Boy Travel
07/29/2015 04:51 pm
thanks for the info, I think you are correct. Although there wasn't a ton of guidance from Google on how to fix the issue I was able to square it away through Yoast's plugin even quicker than editing the file directly myself
John
07/30/2015 07:07 pm
So I have tried numerous configurations of robots.tx to no avail. However, I disabled the wordpress plugin Quick Adsense and all renders OK. Could someone who knows how to code look into the issue and maybe post a fix ? To be more specific this is what showed in the list after I tried to render the page before deactivating Quick Adsense. "http ://page ad2.googlesyndication. com/pagead/js/ adsbygoogle.js " After deactivating Quick Adsense the page rendered OK Edited to break link. *Please note this robots.txt did not solve the problem. My current robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Allow: /wp-content/themes/ Allow: /wp-content/plugins/ Allow: /wp-content/uploads/ Allow: /wp-includes/css/ Allow: /wp-includes/js/ Allow: /wp-includes/images/ User-Agent: Googlebot Allow: .js Allow: .css Allow: /*.js* Allow: /*.css* Allow: /wp-admin/ Sitemap: http://mywebsitename. com/sitemapindex.xml
Schul
07/31/2015 10:33 am
Thanks, i used Fetch as Googlebot / Fetch and i have this result: 0 error*0 warning with this line on red: Disallow: /wp-includes/. So i must allo this file but i can't find the robot.txt on my serveur to edit it! how can i solve this pls? thanks in advane.
Teach Leaks
08/01/2015 02:10 pm
hi! the disallow wp-admin ain't necessary? i am getting the same cs and js warning please help should i delete disallow wp-admin from robots.txt file?
Monte Kivo
08/14/2015 08:32 pm
Is this going to be a problem for everyone?
1andomstar
12/09/2015 02:28 pm
Google wants to make the web look pretty for Chrome. It's a business.
Hemang Rindani
12/21/2015 10:03 am
WordPress is a great enterprise content management system that allows a developer to build websites for any business scenario. The splendid features that comes in the form of modules and plugins can efficiently satisfy any business requirement. For a website to be visible, there are certain things that must be done correctly. The important part of website is proper installation of robots.txt file along with implementation of various SEO tools and applying proper settings. Most of the search engines enters site and scans content without scanning styles and CSS and that is why WordPress' themes are content focused. This error can be resolved by adding proper tags that allows search engines to scan them and enhance the website search results.
Abbas Mehran
01/23/2017 03:16 am
Hi I am new here, and with a problem. Chrome and Google Search Console tell me that my site has robots.txt and that contains User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php But someone somehow has written a robots.txt tand injected somewhere that prevents Home page being indexed. I can't fine this robots.txt, or rather no robots.txt anywhere going to my files through Cpanel. Based on OnPage.org Zoom, report shows this as noindex via robots. Any advice, please