Skip to content

Login
Hypefactors front page

Kestrel, the Hypefactors web crawler

If you see Kestrel in your server logs, this page explains who is crawling your site, why, and how to limit it or get in touch. Its user agent is:

Kestrel/1.0 (Hypefactors; +https://hypefactors.com/kestrel)

Who runs Kestrel

Kestrel is the web crawler of Hypefactors, a media-monitoring service. It collects publicly available news articles so that Hypefactors' customers can follow what is written about them.

What it fetches

  • Public news pages
  • The site's listings of new articles: RSS and Atom feeds, sitemaps, and front and section pages
  • robots.txt

Kestrel does not log in, submit forms, run JavaScript or keep cookies.

How it behaves

  • It obeys the rules in robots.txt for the user agent token Kestrel, and honours Crawl-delay.
  • By default it makes at most one request to each server every 5.5 seconds.
  • It backs off when a site answers with 429 or 403, or refuses access to its robots.txt.
  • It uses conditional requests (ETag and If-Modified-Since), so it does not fetch unchanged pages again.

How to limit or block it

Add rules for Kestrel to your robots.txt. This example keeps it out of /private/ and asks for at least 10 seconds between requests:

User-agent: Kestrel
Disallow: /private/
Crawl-delay: 10

To block Kestrel from the whole site:

User-agent: Kestrel
Disallow: /

Where requests come from

Kestrel sends its requests from these IP addresses:

  • 35.228.87.242
  • 35.228.235.236

The addresses can change. This page lists the current ones.

Contact

If you have questions or complaints about Kestrel, or want to make an arrangement for your site, email [email protected].