This iThemes Security Setting Can Block Search Engine Bots!

We were recently beset by a sudden, MASSIVE drop in the number of search referrals from Google. We discovered that something was blocking search engine robots from accessing our sitemaps or crawling our website!

At first, we though it was due to a CloudFlare SSL configuration. But now, we have confirmed that it is due to an iThemes Security feature!

It is critical that you check and make sure that this iThemes Security setting is not preventing search engine robots from reading your sitemaps, and crawling your website.

 

iThemes Security

Formerly known as Better WP Security, iThemes Security is a very popular WordPress security plugin. While the paid Pro option offers a ton of cool security features, even the free plugin gives you the ability to lock down WordPress, fix common holes, stop automated attacks and implement a blacklist (this is a pivotal feature in this article).

This iThemes Security Setting Can Block Search Engine Bots!

 

What Happens If You Block Search Engine Bots?

if you block search engine bots accidentally or otherwise, you prevent them from indexing your website. This essentially makes your website “invisible” to search engines. Your website pages will no longer appear when people are searching for a relevant topic.

How does that happen?

  • Your website pages no longer appear in Google searches. If Google can’t see them, Google cannot display your pages in search results!
  • Even if your website pages do appear in Google searches, the links may be corrupt or nonsensical. Look at this example of this search result which leads to a bad link.

How To Fix Sitemap Problems + Reindex Your Website - Google Search error

  • The description of your website page may also be nonsensical, as the example above also demonstrates.

 

This iThemes Security Setting Can Block Search Engine Bots!

A key feature of iThemes Security is the ability to set up a blacklist. iThemes Security will automatically populate the blacklist with the IP addresses after a number of failed attempts to login. This prevents a malicious attacker from trying to brute force its way into your system.

This iThemes Security Setting Can Block Search Engine Bots!

Unfortunately, it can falsely detect search engine bots as malicious hackers, and add them to the blacklist. That was precisely what happened to us.

Google reports that all of our sitemaps are inaccessible. Here is a view showing all five of our sitemaps were inaccessible.

How To Detect + Fix Sitemap Problems In Google Search Console - Checking for sitemap errors

If you click on the reported errors, they will all show HTTP 403 error (Forbidden).

How To Detect + Fix Sitemap Problems In Google Search Console - Checking for sitemap errors

You can verify if any search engine bot is being blocked by keying in the sitemap (or robot.txt or your website) at Redirect Checker. You can also try loading the sitemap or robot.txt in your own web browser.

If Redirect Checker or you have no problem accessing your sitemaps or robot.txt file, then something is blocking the Googlebot (or other search engine robots) from accessing your sitemaps, or crawling your website. That “something” is most likely iThemes Security’s blacklist.

 

The Solution

The solution is simple.

  1. Log into your website’s WordPress admin panel.
  2. Go to Security -> Settings.
  3. Look for the Banned Users section, and click on Configure Settings.

This iThemes Security Setting Can Block Search Engine Bots!

  1. In the Banned Users page, you will see a list of banned IP addresses.
  2. Delete the whole list of banned IP addresses.

This iThemes Security Setting Can Block Search Engine Bots!

  1. Uncheck the Ban Lists option.

This iThemes Security Setting Can Block Search Engine Bots!

  • Click Save Settings, and that’s it! The search engine robots will now be able to read your sitemaps!

 

Pro Tips

For those who still want to use the blacklist to block malicious attackers, here are the IP addresses used by various search engine robots.

Google will eventually read your sitemaps and reindex your website. But you can speed things along by :

Both guides were written with help from Kok Kee from Nasi Lemak Tech!

 

Suggested Reading

Go Back To > Articles | Home

 

Support Tech ARP!

If you like our work, you can help support our work by visiting our sponsors, participating in the Tech ARP Forums, or even donating to our fund. Any help you can render is greatly appreciated!

Comments

comments

About The Author

Related posts

Leave a Reply

%d bloggers like this: