In a recent Business Bloomer Club Slack thread, a member sought lightweight solutions to block unwanted site crawlers without using bulky security plugins like Wordfence. This sparked a discussion about effective ways to combat bot traffic on WooCommerce stores, which can significantly impact website performance and security if left unchecked.
Several approaches emerged, ranging from server-side configurations to specialized plugins. Server-side rules in .htaccess or web.config offer direct control, while lightweight plugins can simplify the process for users less comfortable with manual coding. Managing bot traffic helps ensure smoother site operations, better resource management, and enhanced user experiences.
Let’s explore these options, including practical examples of server-side rules and plugin recommendations, to protect your WooCommerce store from excessive bot activity without unnecessary overhead.
Using .htaccess to Block Bots on Linux Servers
One effective method shared in the discussion involves configuring server rules via the .htaccess file. This file can specify which user agents (bots) to block:
SetEnvIfNoCase User-Agent .*unwanted_bot.* unwanted_bot
Deny from env=unwanted_bot
By including patterns for specific bot names such as bingbot or commoncrawl, you can deny their access to your site. For more flexibility, you can customize the list to include additional unwanted bots.
Example .htaccess Block List
SetEnvIfNoCase User-Agent .*gptbot.* unwanted_bot
SetEnvIfNoCase User-Agent .*ahrefs.* unwanted_bot
SetEnvIfNoCase User-Agent .*twitterbot.* unwanted_bot
Deny from env=unwanted_bot
This lightweight approach helps block common and AI-based bots without installing extra plugins.
Blocking Bots with web.config on Windows Servers
For sites running on Windows servers, the equivalent configuration resides in the web.config file:
<rule name="BlockUserAgents" patternSyntax="Wildcard" stopProcessing="true">
<match url="*" />
<conditions logicalGrouping="MatchAny">
<add input="{HTTP_USER_AGENT}" pattern="*gptbot*" />
<add input="{HTTP_USER_AGENT}" pattern="*ahrefs*" />
<add input="{HTTP_USER_AGENT}" pattern="*bingbot*" />
</conditions>
<action type="AbortRequest" />
</rule>
This configuration allows you to block user agents by defining rules based on the bot patterns you want to restrict. It provides a lightweight solution for improving site security and performance.
Plugin Alternatives for Blocking Bots
If manual configuration isn’t preferable, free plugins are available:
- Stop and Block Bots: Provides simple settings for bot management. [WordPress repo]
- Blackhole for Bad Bots: Traps bots in a hidden link to prevent further crawling. [WordPress repo]
- Block AI Crawlers: A GitHub solution targeting AI bots like GPT. [GitHub]
These plugins offer varying degrees of bot management without the overhead of larger security suites.
Conclusion
Fighting unwanted bots requires a balance between server-side efficiency and ease of use. Depending on your technical comfort level, either server rule configurations or lightweight plugins can effectively manage bot traffic on your WooCommerce site.
