Recently a lot of websites have been closing due to competition in the digital marketing world. With the market becoming so competitive, it is important for companies to be able to monitor and understand their audience as much as possible to not only compete but also survive.
There is an article about crawler, please watch it together. If you have any questions, remember to reply.
A few users have been curious about how typically the crawler data upon the crawler-aware site is organized, now we will end up being more than curious to reveal just how the crawler information is collected and organized.

We may reverse the IP address of the crawler to query typically the rDNS, such as: we find this IP: 116. 179. thirty-two. 160, rDNS simply by reverse DNS lookup tool: baiduspider-116-179-32-160. spider. baidu. com

To sum up, we can roughly determine should be Baidu google search bots. Because Hostname can be forged, and we only reverse search, still not accurate. We also want to forward search, we ping command to find baiduspider-116-179-32-160. crawl. baidu. apresentando could be resolved as: 116. 179. thirty-two. 160, through the particular following chart can be seen baiduspider-116-179-32-160. crawl. baidu. apresentando is resolved in order to the Internet protocol address 116. 179. 32. one hundred sixty, which means that will the Baidu search engine crawler is usually sure.

Searching simply by ASN-related information

Not all crawlers follow typically the above rules, the majority of crawlers reverse lookup without any effects, we need in order to query the IP address ASN information to determine when the crawler info is correct.

For instance , this IP is usually 74. 119. 118. 20, we may see that this IP address is typically the Internet protocol address of Sunnyvale, California, USA simply by querying the IP information.

We can see by typically the ASN information that will he is an IP of Criteo Corp.

The screenshot previously mentioned shows the signing information of critieo crawler, the yellow part is their User-agent, accompanied by its IP, and absolutely nothing wrong with this admittance (the IP will be indeed the Internet protocol address of CriteoBot).

Internet protocol address segment published from the crawler’s official documentation

Some crawlers post IP address sections, and we save typically the officially published IP address segments associated with the crawler directly to the database, which can be an easy plus fast way to do this.

By means of public logs

We are able to often view public logs on typically the Internet, for instance , the following image is really a public log record I found.

We all can parse the particular log records to determine which are crawlers and which often are visitors dependent on the User-agent, which greatly enriches our database regarding crawler records.

Overview

The above four procedures detail how the particular crawler identification website collects and organizes crawler data, plus how to ensure the accuracy in addition to reliability of the particular crawler data, but of course right now there are not just the above four procedures in the real operation process, but they are fewer used, so these people are not introduced in this article.