By Alexander Njogu in nginx — May 16, 2021

Blocking bad bots and restricting users based on geographical location on GCP+Firebase using NGINX

Security is not easy and you can never be 100% full proof however you can take measures. These measures can be helpful in reducing your cloud billing costs and further more protect you from hacks...

Photo from cloudiqtech

As a web admin, the internet is very funny place you get to see all kind of traffic. From Russian bots trying to execute wp-admin page to China bots trying to find JavaScript vulnerabilities.

Luckily GCP logs all kinds of traffic and your can analyses your site traffic at a letter time.

The site is not WordPress based and the hacker got away with 404 error however this is unnecessary traffic from hacker bots.

Its natural to want to protect your website from unwanted traffic, especially to users in a geographical location where the site doesn't really offer any meaningful value.

You would think well since Firebase is your hosting provider they would provide you with some sort of user panel or some way to secure your site from bad bots or geographical restriction. Unfortunately Firebase just offers you basic DDOS protection. I find Cloudflare having more user friendly features to protect websites from attacks than Firebase which doesn’t have a security panel at all.

You might ask how about we use Firebase with Cloudflare. That’s a big NO. If you try that your website will suffers huge latency problems. As engineers its not in our place to whine about problems but finding solutions. The solution is relying on nginx to block bad bots and performing geographical restrictions. Its not all that straightforward so pay attention.

Assumption: Your running your infrastructure on GCP cloud run + Firebase however most methods should work on other cloud providers.

Blocking Bad Bots based on user agents NGINX

This is the simplest layer of protection in nginx. It prevents bots from overwhelming your server after you detect them.

Blocking user agents containing certain words in user agent.

Edit nginx’s : /etc/nginx/conf.d/default.conf

server {
....

if ($http_user_agent ~* (wget|curl)) {
    return 403;
}
...
}

This blocks any user agent containing the words wget or curl so you should be careful with this. Its also case insensitive, if you want case sensitivity remove the asterisk (*).

Lets say SEMrush is overwhelming your server with request.
Its list of user agents look as follows:

Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)

Adding SemrushBot in your block list would block all these bots returning 403 error to the bots.

server {
....

if ($http_user_agent ~* (wget|curl|SemrushBot)) {
    return 403;
}
...
}.

If you have a long list to block you can use the nginx map directive to block bots. This is a lot cleaner and saves your from going insane when you look at your own configurations.

map $http_user_agent $badagent {
        default         0;
        ~*SemrushBot    1;
        ~*wget          1;
        ~curl           1; # case sensitive
}

server {
......

if ($badagent) {
        return 403;
    }
......

}

Blocking exact User Agents

Consider a bot that uses a user agent such us Mozilla/5.0 . This user agent contains common phrases found on legitimate browser. Blocking Mozilla would deny legitimate users from using your site. This is where blocking exact user agents is necessary. To block exact user agents we use the equals sign.

server {
....

if ($http_user_agent = "Mozilla/5.0"){
    return 403;
}
...
}

Restricting Users Based on Geographical Location NGINX

Installing NGINX GEOIP module

The GEOIP module can be used for many things such as selecting the nearest server for clients for faster responses. However in our case we just want to use it to block users based on their geographical location. For more i recommend reading the nginx documentation:
https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-by-geoip/

For the sake of simplicity lets assume our cloud container is of Debian distribution specifically Ubuntu.
For more on installing GEOIP on different distribution check the nginx documentation here.

Installing GEOIP on Debian/Ubuntu

We first install the necessary dependencies.

apt-get install geoip-database libgeoip1

Loading the GEOIP module

Edit: /etc/nginx/nginx.conf

This is just loads the necessary module.

load_module "modules/ngx_http_geoip_module.so"

Blocking users Geographically

Edit: /etc/nginx/conf.d/default.conf

geoip_country /usr/share/GeoIP/GeoIP.dat; # Load IP database
map $geoip_country_code $allowed_country { 
    default yes;
     RU no; # block Russia
     CN no; # block china
}
server {

.....

}

As far as nginx is concerned that's all the setting required to block users based on geographical location. If nginx was not behind anything this would work perfectly. If your nginx is not behind a load balancer this is where you stop.

Make GEOIP work behind load balancer / Get client’s real IP

The problem at hand when nginx is behind a load balancer is that the real IP of the client does not get to nginx. If you analyze the nginx logs you’ll find out the IP logged belongs to the cloud platform.

Getting Client IP on Google Cloud LB

In order to get clients IP behind a load balancer we do the following configurations.