Recently, I changed the hosting company for my website http://bazaarlive.info and I noticed that Google webmaster’s tool central area was showing error in the sitemap of my website.
If you are also getting “Network unreachable or robots.txt not reachable” error message in the Google Webmaster’s area, then you can take it for sure that there is some grave problem with your host’s server and not a glitch in google system.
I googled about this problem and noticed that this is a very common error but no solution has been suggested anywhere. So, I thought of writing here.
Usually this problem arises when host server blocks google bot from accessing your website by denying access to google IPs in the server firewall..
Google tries to ping the server for the robots.txt file or sitemap.xml but because the IP was denied access in the iptables, googlebots are not able to access the robots.txt file and hence the problem.
So here is what one must do to fix this error:-
1. Do iptables-save &> /home/username/temp-ip-tables.txt
Cross check all IP addresses in the /home/username/temp-ip-tables.txt file using arin.net/whois and note if any of those ips belong to googlebot. If any ip (usually 66.249.x.x) belongs to google then make sure you remove it from iptables (csf.deny file) which can vary depending on the firewall you/your host are using.
To whitelist Google, you can whitelist 66.249.*.*
IPs of other important search engines are:-
Allow from 127.0.0.1 Allow from 65.55 # MSN Allow from 66.249 # Google Allow from 67.195 # Yahoo! Allow from 72.30 # Yahoo! Allow from 74.6 # Yahoo! Allow from 122.152.129.15 # Baidu
I hope this post will help many people.
