Tailbot

Tailbot

Tailbot is Tail’s web crawling bot. Crawling is the process by which Tailbot reads and understand the content of a web page to provide several features to our partners.

Tail uses a set of computers to crawl billions of web pages. The number of pages crawled and which computer will crawl a website will be decided by an algorithm.

For webmasters:

Usually, Tailbot will not crawl your pages more than once every few seconds. However, if your website joined Tail network recently, you may see a larger rate of accesses coming from our servers in the first weeks.

Each webpage is crawled by Tailbot only once.

The IP addresses used by Tailbot change from time to time. The best way to identify accesses by Tailbot is to use the user-agent (Tailbot).

Here is a list of agents used by Tailbot:

  • • Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2 (compatible; Tailbot/1.0; +http://www.tailtarget.com/tailbot/)

Tailbot IPs API:

 

You can also periodically call our Tailbot IPs API to obtain a list of IPs currently used by Tailbot. Since this list can change anytime, please, make sure you call the API periodically to refresh the list of IPs. This is the Tailbot IPs API URL:

https://api.tailtarget.com/api/v2/open/data/resources/crawlers/ips