How to Block Unwanted User-Agent & Referrers in Apache, Nginx and WordPress?

Block all these pointless requests you see within the logs

I assume you’ve got checked your web site’s customer logs and have had sufficient of these undesirable requests. They don’t seem to be essentially unhealthy or spammy, however they could not add worth to the corporate. Think about that almost all of hits come by way of that undesirable consumer agent or referrers and also you suppose your web site is getting good visitors however in actuality they’re ineffective.

One of the best ways to handle them is to cease them on the edge, reminiscent of community gadgets, load balancer, firewall, or CDN. However I perceive that it is probably not possible for a private blogger or small web sites to make use of, and you could need to block at a decrease degree, reminiscent of net servers, WordPress, and so forth.

I hope you have already got a listing of referrer and consumer brokers you need to block. Let’s begin.

As a greatest apply, again up the configuration file earlier than making any adjustments so as to roll again if one thing goes improper.

Nginx

Nginx helps hundreds of thousands of websites and may be very fashionable amongst internet hosting. If you’re utilizing Nginx, this is the best way to cease it. As an instance you get lots of automated requests to the following consumer agent and also you determined to dam them.

• Java
• Curl
• Python

if ($http_user_agent ~* "java|curl|python") {
    return 403;
}

In order for you these to be redirected someplace, do the next:

if ($http_user_agent ~* "java|curl|python") {
    return 301 https://yoursite.com;
}

The above configuration should be below the server block.

And the following to dam by referrers. The next instance, which is below the location block for blocking requests from semalt.com, badsite.web, instance.com.

if ($http_referer ~ "semalt.com|badsite.web|instance.com")  {
  return 403;
}

After making the mandatory adjustments, you will need to save the file and restart Nginx for results to take impact.

To restart Nginx you need to use:

service nginx restart

Nginx is a robust net server and if you happen to’re fascinated by studying, try this on-line course.

Apache HTTP

To dam consumer agent in Apache you need to use the mod_rewrite module. Be sure the module is enabled after which add the next .htaccess file or respectively .conf file.

When you have a number of websites configured and need to block them for a selected URL, you could need to put them of their respective websites VirtualHost part.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} badcrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} badbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} badspider [NC]
RewriteRule . - [R=403,L]

The above rule will block any request containing user-agent as badcrawler, badbot and badspider.

And the instance under to dam primarily based on referrer title BlowFish, CatchBot, BecomeBot.

RewriteEngine on
RewriteCond %{HTTP_REFERER} blowfish|CatchBot|BecomeBot [NC]
RewriteRule . - [R=403,L]

Restart the Apache server as typical and take a look at the outcomes.

WordPress

If you’re utilizing WordPress on shared internet hosting or haven’t got entry to the net server configuration or do not feel comfy altering the file then you need to use the WP plugin. There are numerous WP safety plugins, and one of many fashionable ones for blocking unhealthy bots is Blackhole for Dangerous Bots.

Conclusion

I hope the above suggestions assist you to cease the unhealthy suggestions in order that legit requests usually are not affected. If you’re on the lookout for complete safety safety, it’s also possible to think about using cloud-based WAF reminiscent of Astra or SUCURI.