Brief crawler

Brief is the web crawler run by Tekodot Ltd to build a search index. If you've seen it in your logs, this page explains what it does and how to control it.

How to recognise it

It sends this user agent:

Mozilla/5.0 (compatible; Brief/1.0; +https://crawl.prebrief.co/)

Its robots.txt name is Brief.

How it behaves

• It reads and obeys robots.txt, including Crawl-delay.
• It waits between requests to the same site and keeps very few connections open to any one site.
• It slows down or stops when a site returns 429 or 503, and honours Retry-After.
• It uses conditional requests, so pages that haven't changed aren't downloaded again.
• It does not log in, fill in forms or try to get past blocks or bot checks.

How to block or slow it

To block it from your whole site, add this to your robots.txt:

User-agent: Brief
Disallow: /

To slow it down instead, set a delay in seconds between requests:

User-agent: Brief
Crawl-delay: 10

Changes to robots.txt may take up to a day to be picked up. If you need it to stop sooner, email us.

Contact

If the crawler is causing problems for your site, email luke@tekodot.com. Please include your domain and, if you can, the time of the requests.