Crawlers of Clarabot

Clarabot Zrt operates own web crawlers; also called bots. We do so to be independent and be able to ensure our data is trustworthy.

Our company and our crawlers share the same name: Clarabot

How Clarabot visits your site

Most of the time, Clarabot will not make more than one request every few seconds to individual sites. The goal of our crawler is to visit as many pages as possible, without hindering the hosting server’s bandwidth and performance.

Example user-agent values that we use:

Mozilla/5.0 (compatible; Clarabot/1.0; +http://www.clarabot.info/bots)
Mozilla/5.0 (compatible; Clarabot/1.4; +http://www.clarabot.info/bots)