type
Post
status
Published
date
Nov 18, 2022
summary
tags
干货
category
外贸
slug
icon
password
“只有同行之间才是赤裸裸的仇恨。”——郭德纲
- 为什么要屏蔽我标题里提及的那些机器人?那些机器人都是干什么的?
Semrush、Ahrefs、Moz和Majestic等海外SEO软件爬虫机器人存在的目的是抓取你的网站数据,给软件使用者提供数据参考,然后就是如何超越你。拿SEMrush举例,我先打马赛克给大家看一下,有多可怕。
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2F2436a59a-6143-491a-b404-8ffe845112da%2F16b7ff27-f9c4-4f6e-9687-411d4fa22bed%2FUntitled.png%3Fid%3D0446c00e-e53d-40a8-b360-1ee653f7dfae%26table%3Dblock%26spaceId%3D2436a59a-6143-491a-b404-8ffe845112da%26expirationTimestamp%3D1722139200000%26signature%3Dr4FlWNDT9VL7rZ9E1tKPrSy4EwMy2y9kiwP_pwPd0wA?table=block&id=0446c00e-e53d-40a8-b360-1ee653f7dfae&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2F2436a59a-6143-491a-b404-8ffe845112da%2Fbbdbb2f2-0afd-4be3-9c93-6edc7701aa23%2FUntitled.png%3Fid%3Df44c5bfe-cc46-46de-8ba4-a912ca6fb7b2%26table%3Dblock%26spaceId%3D2436a59a-6143-491a-b404-8ffe845112da%26expirationTimestamp%3D1722139200000%26signature%3DH5u9CAbCHoLl3usdb2L7ZqgaVJE_Q8AK6RruDUxSumU?table=block&id=f44c5bfe-cc46-46de-8ba4-a912ca6fb7b2&cache=v2)
上图中,图1可以快速知道你的网站给你带来流量的关键词是什么,图2可以快速知道你的最新反链来自哪里,这是很可怕的。SEO最重要的就是内容和外链了,这一下全被有心人监听了个彻底。
📝那么,如何屏蔽这些SEO软件爬虫呢?
robots.txt文件中屏蔽
User-agent: Rogerbot User-agent: Exabot User-agent: MJ12bot User-agent: Dotbot User-agent: Gigabot User-agent: AhrefsBot User-agent: SemrushBot User-agent: SemrushBot-SA Disallow: /
在.htaccess中屏蔽,添加以下代码:
<IfModule mod_rewrite.c> RewriteEngine on RewriteCond %{HTTP_USER_AGENT} (ahrefsbot|mj12bot|rogerbot|exabot|dotbot|gigabot|semrush) [NC] RewriteRule .* - [F,L] </IfModule>
上面俩个方法基本上最有效,第一个防君子,第二个防小人。俩个一起用就可以了,如果你觉得不踏实,可以看一下我下面列出的原文出处。
原文出处:https://www.wisamabdulaziz.com/how-to-block-link-cralwers-like-majestic-ahrefs-moz-and-semrush.htm (致谢)