Linux系统入门学习:Nginx网络服务器上如何阻止特定用户代理(UA
问题: 我注意到有一些机器人经常访问我的Nginx驱动的网站,并且进行一些攻击性的扫描,导致消耗掉了我的网络服务器的大量资源。我一直尝试着通过用户代理符串来阻挡这些机器人。我怎样才能在Nginx网络服务器上阻挡掉特定的用户代理呢?
现代互联网滋生了大量各种各样的恶意机器人和网络爬虫,比如像恶意软件机器人、垃圾邮件程序或内容刮刀,这些恶意工具一直偷偷摸摸地扫描你的网站,干些诸如检测潜在网站漏洞、收获电子邮件地址,或者只是从你的网站偷取内容。大多数机器人能够通过它们的“用户代理”签名字符串来识别。
作为第一道防线,你可以尝试通过将这些机器人的用户代理字符串添加入robots.txt文件来阻止这些恶意软件机器人访问你的网站。但是,很不幸的是,该操作只针对那些“行为良好”的机器人,这些机器人被设计遵循robots.txt的规范。许多恶意软件机器人可以很容易地忽略掉robots.txt,然后随意扫描你的网站。
另一个用以阻挡特定机器人的途径,就是配置你的网络服务器,通过特定的用户代理字符串拒绝要求提供内容的请求。本文就是说明如何在nginx网络服务器上阻挡特定的用户代理。
在Nginx中将特定用户代理列入黑名单
要配置用户代理阻挡列表,请打开你的网站的nginx配置文件,找到server
定义部分。该文件可能会放在不同的地方,这取决于你的nginx配置或Linux版本(如,/etc/nginx/nginx.conf
,/etc/nginx/sites-enabled/<your-site>
,/usr/local/nginx/conf/nginx.conf
,/etc/nginx/conf.d/<your-site>
)。
<span class="pln">server </span><span class="pun">{</span>
<span class="pln">listen </span><span class="lit">80</span><span class="pln"> default_server</span><span class="pun">;</span>
<span class="pln">server_name xmodulo</span><span class="pun">.</span><span class="pln">com</span><span class="pun">;</span>
<span class="pln">root </span><span class="pun">/</span><span class="pln">usr</span><span class="pun">/</span><span class="pln">share</span><span class="pun">/</span><span class="pln">nginx</span><span class="pun">/</span><span class="pln">html</span><span class="pun">;</span>
<span class="pun">....</span>
<span class="pun">}</span>
在打开该配置文件并找到 server
部分后,添加以下 if 声明到该部分内的某个地方。
<span class="pln">server </span><span class="pun">{</span>
<span class="pln">listen </span><span class="lit">80</span><span class="pln"> default_server</span><span class="pun">;</span>
<span class="pln">server_name xmodulo</span><span class="pun">.</span><span class="pln">com</span><span class="pun">;</span>
<span class="pln">root </span><span class="pun">/</span><span class="pln">usr</span><span class="pun">/</span><span class="pln">share</span><span class="pun">/</span><span class="pln">nginx</span><span class="pun">/</span><span class="pln">html</span><span class="pun">;</span>
<span class="com"># 大小写敏感的匹配</span>
<span class="kwd">if</span><span class="pun">(</span><span class="pln">$http_user_agent </span><span class="pun">~</span><span class="pun">(</span><span class="typ">Antivirx</span><span class="pun">|</span><span class="typ">Arian</span><span class="pun">)</span><span class="pun">{</span>
<span class="kwd">return</span><span class="lit">403</span><span class="pun">;</span>
<span class="pun">}</span>
<span class="com">#大小写无关的匹配</span>
<span class="kwd">if</span><span class="pun">(</span><span class="pln">$http_user_agent </span><span class="pun">~*</span><span class="pun">(</span><span class="pln">netcrawl</span><span class="pun">|</span><span class="pln">npbot</span><span class="pun">|</span><span class="pln">malicious</span><span class="pun">))</span><span class="pun">{</span>
<span class="kwd">return</span><span class="lit">403</span><span class="pun">;</span>
<span class="pun">}</span>
<span class="pun">....</span>
<span class="pun">}</span>
如你所想,这些 if 声明使用正则表达式匹配了任意不良用户字符串,并向匹配的对象返回403 HTTP状态码。 $http_user_agent
是HTTP请求中的一个包含有用户代理字符串的变量。‘~’操作符针对用户代理字符串进行大小写敏感匹配,而‘~*’操作符则进行大小写无关匹配。‘|’操作符是逻辑或,因此,你可以在 if 声明中放入众多的用户代理关键字,然后将它们全部阻挡掉。
在修改配置文件后,你必须重新加载nginx以激活阻挡:
<span class="pln">$ sudo </span><span class="pun">/</span><span class="pln">path</span><span class="pun">/</span><span class="pln">to</span><span class="pun">/</span><span class="pln">nginx </span><span class="pun">-</span><span class="pln">s reload</span>
你可以通过使用带有 “--user-agent” 选项的 wget 测试用户代理阻挡。
<span class="pln">$ wget </span><span class="pun">--</span><span class="pln">user</span><span class="pun">-</span><span class="pln">agent </span><span class="str">"malicious bot"</span><span class="pln"> http</span><span class="pun">:</span><span class="com">//<nginx-ip-address></span>
管理Nginx中的用户代理黑名单
目前为止,我已经展示了在nginx中如何阻挡一些用户代理的HTTP请求。如果你有许多不同类型的网络爬虫机器人要阻挡,又该怎么办呢?
由于用户代理黑名单会增长得很大,所以将它们放在nginx的server部分不是个好点子。取而代之的是,你可以创建一个独立的文件,在该文件中列出所有被阻挡的用户代理。例如,让我们创建/etc/nginx/useragent.rules,并定义以下面的格式定义所有被阻挡的用户代理的图谱。
<span class="pln">$ sudo vi </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">nginx</span><span class="pun">/</span><span class="pln">useragent</span><span class="pun">.</span><span class="pln">rules</span>
<span class="pln">map $http_user_agent $badagent </span><span class="pun">{</span>
<span class="kwd">default</span><span class="lit">0</span><span class="pun">;</span>
<span class="pun">~*</span><span class="pln">malicious </span><span class="lit">1</span><span class="pun">;</span>
<span class="pun">~*</span><span class="pln">backdoor </span><span class="lit">1</span><span class="pun">;</span>
<span class="pun">~*</span><span class="pln">netcrawler </span><span class="lit">1</span><span class="pun">;</span>
<span class="pun">~</span><span class="typ">Antivirx</span><span class="lit">1</span><span class="pun">;</span>
<span class="pun">~</span><span class="typ">Arian</span><span class="lit">1</span><span class="pun">;</span>
<span class="pun">~</span><span class="pln">webbandit </span><span class="lit">1</span><span class="pun">;</span>
<span class="pun">}</span>
与先前的配置类似,‘~*’将匹配以大小写不敏感的方式匹配关键字,而‘~’将使用大小写敏感的正则表达式匹配关键字。“default 0”行所表达的意思是,任何其它文件中未被列出的用户代理将被允许。
接下来,打开你的网站的nginx配置文件,找到里面包含 http 的部分,然后添加以下行到 http 部分某个位置。
<span class="pln">http </span><span class="pun">{</span>
<span class="pun">.....</span>
<span class="pln">include </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">nginx</span><span class="pun">/</span><span class="pln">useragent</span><span class="pun">.</span><span class="pln">rules</span>
<span class="pun">}</span>
注意,该 include 声明必须出现在 server 部分之前(这就是为什么我们将它添加到了 http 部分里)。
现在,打开nginx配置定义你的服务器的部分,添加以下 if 声明:
<span class="pln">server </span><span class="pun">{</span>
<span class="pun">....</span>
<span class="kwd">if</span><span class="pun">(</span><span class="pln">$badagent</span><span class="pun">)</span><span class="pun">{</span>
<span class="kwd">return</span><span class="lit">403</span><span class="pun">;</span>
<span class="pun">}</span>
<span class="pun">....</span>
<span class="pun">}</span>
最后,重新加载nginx。
<span class="pln">$ sudo </span><span class="pun">/</span><span class="pln">path</span><span class="pun">/</span><span class="pln">to</span><span class="pun">/</span><span class="pln">nginx </span><span class="pun">-</span><span class="pln">s reload</span>
现在,任何包含有/etc/nginx/useragent.rules
中列出的关键字的用户代理将被nginx自动禁止。
Nginx 的详细介绍:请点这里
Nginx 的下载地址:请点这里
via: http://ask.xmodulo.com/block-specific-user-agents-nginx-web-server.html