We were recently beset by a sudden, MASSIVE drop in the number of search referrals from Google. We discovered that something was blocking search engine robots from accessing our sitemaps or crawling our website!
At first, we though it was due to a CloudFlare SSL configuration. But now, we have confirmed that it is due to an iThemes Security feature!
It is critical that you check and make sure that this iThemes Security setting is not preventing search engine robots from reading your sitemaps, and crawling your website.
iThemes Security
Formerly known as Better WP Security, iThemes Security is a very popular WordPress security plugin. While the paid Pro option offers a ton of cool security features, even the free plugin gives you the ability to lock down WordPress, fix common holes, stop automated attacks and implement a blacklist (this is a pivotal feature in this article).
What Happens If You Block Search Engine Bots?
if you block search engine bots accidentally or otherwise, you prevent them from indexing your website. This essentially makes your website “invisible” to search engines. Your website pages will no longer appear when people are searching for a relevant topic.
How does that happen?
- Your website pages no longer appear in Google searches. If Google can’t see them, Google cannot display your pages in search results!
- Even if your website pages do appear in Google searches, the links may be corrupt or nonsensical. Look at this example of this search result which leads to a bad link.
- The description of your website page may also be nonsensical, as the example above also demonstrates.
This iThemes Security Setting Can Block Search Engine Bots!
A key feature of iThemes Security is the ability to set up a blacklist. iThemes Security will automatically populate the blacklist with the IP addresses after a number of failed attempts to login. This prevents a malicious attacker from trying to brute force its way into your system.
Unfortunately, it can falsely detect search engine bots as malicious hackers, and add them to the blacklist. That was precisely what happened to us.
Google reports that all of our sitemaps are inaccessible. Here is a view showing all five of our sitemaps were inaccessible.
If you click on the reported errors, they will all show HTTP 403 error (Forbidden).
You can verify if any search engine bot is being blocked by keying in the sitemap (or robot.txt or your website) at Redirect Checker. You can also try loading the sitemap or robot.txt in your own web browser.
If Redirect Checker or you have no problem accessing your sitemaps or robot.txt file, then something is blocking the Googlebot (or other search engine robots) from accessing your sitemaps, or crawling your website. That “something” is most likely iThemes Security’s blacklist.
The Solution
The solution is simple.
- Log into your website’s WordPress admin panel.
- Go to Security -> Settings.
- Look for the Banned Users section, and click on Configure Settings.
- In the Banned Users page, you will see a list of banned IP addresses.
- Delete the whole list of banned IP addresses.
- Uncheck the Ban Lists option.
- Click Save Settings, and that’s it! The search engine robots will now be able to read your sitemaps!
Pro Tips
[adrotate group=”2″]For those who still want to use the blacklist to block malicious attackers, here are the IP addresses used by various search engine robots.
Google will eventually read your sitemaps and reindex your website. But you can speed things along by :
Both guides were written with help from Kok Kee from Nasi Lemak Tech!
Suggested Reading
- How To Detect + Fix Sitemap Problems In Google Search Console
- How To Reindex Your Website Using Google Search Console
- Search Engine Robot IP Addresses – Googlebot, BingBot, MSNBot + More!
Go Back To > Articles | Home
Support Tech ARP!
If you like our work, you can help support our work by visiting our sponsors, participating in the Tech ARP Forums, or even donating to our fund. Any help you can render is greatly appreciated!
