[{"data":1,"prerenderedAt":116},["ShallowReactive",2],{"page:\u002Fkestrel\u002F":3,"built-pages":22,"post:\u002Fkestrel\u002F":115},{"page":4,"lang":5,"path":6,"title":7,"description":8,"image":-1,"noindex":9,"sections":10,"testimonials":20,"posts":21},"kestrel","en","\u002Fkestrel\u002F","Kestrel, the Hypefactors web crawler","Kestrel is Hypefactors' web crawler for public news. What it fetches, how it behaves, how to limit or block it in robots.txt, its IP addresses and contact.",false,[11,17],{"type":12,"height":13,"heading":7,"background":14},"hero","medium",{"image":15,"overlay":16},"\u002Fuploads\u002F2019\u002F07\u002Fhypefactors_ai_2.jpg","dark",{"type":18,"body":19},"prose","If you see Kestrel in your server logs, this page explains who is crawling your site, why, and how to limit it or get in touch. Its user agent is:\n\n```\nKestrel\u002F1.0 (Hypefactors; +https:\u002F\u002Fhypefactors.com\u002Fkestrel)\n```\n\n### Who runs Kestrel\n\nKestrel is the web crawler of Hypefactors, a media-monitoring service. It collects publicly available news articles so that Hypefactors' customers can follow what is written about them.\n\n### What it fetches\n\n- Public news pages\n- The site's listings of new articles: RSS and Atom feeds, sitemaps, and front and section pages\n- `robots.txt`\n\nKestrel does not log in, submit forms, run JavaScript or keep cookies.\n\n### How it behaves\n\n- It obeys the rules in `robots.txt` for the user agent token `Kestrel`, and honours `Crawl-delay`.\n- By default it makes at most one request to each server every 5.5 seconds.\n- It backs off when a site answers with `429` or `403`, or refuses access to its `robots.txt`.\n- It uses conditional requests (`ETag` and `If-Modified-Since`), so it does not fetch unchanged pages again.\n\n### How to limit or block it\n\nAdd rules for `Kestrel` to your `robots.txt`. This example keeps it out of `\u002Fprivate\u002F` and asks for at least 10 seconds between requests:\n\n```\nUser-agent: Kestrel\nDisallow: \u002Fprivate\u002F\nCrawl-delay: 10\n```\n\nTo block Kestrel from the whole site:\n\n```\nUser-agent: Kestrel\nDisallow: \u002F\n```\n\n### Where requests come from\n\nKestrel sends its requests from these IP addresses:\n\n- `35.228.87.242`\n- `35.228.235.236`\n\nThe addresses can change. This page lists the current ones.\n\n### Contact\n\nIf you have questions or complaints about Kestrel, or want to make an arrangement for your site, email [cc@hypefactors.com](mailto:cc@hypefactors.com).\n",{},[],[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114],"_home\u002Fda","_home\u002Fde","_home\u002Fen","_home\u002Fno","_home\u002Fsv","about\u002Fda","about\u002Fde","about\u002Fen","about\u002Fno","about\u002Fsv","agency-landingpage\u002Fda","agency-landingpage\u002Fde","agency-landingpage\u002Fen","agency-landingpage\u002Fno","agency-landingpage\u002Fsv","arts-education\u002Fen","blog\u002Fda","blog\u002Fen","book-a-free-demo\u002Fda","book-a-free-demo\u002Fde","book-a-free-demo\u002Fen","book-a-free-demo\u002Fno","book-a-free-demo\u002Fsv","cancellation-and-refund-policy\u002Fen","cookie-declaration\u002Fen","create\u002Fda","create\u002Fde","create\u002Fen","create\u002Fno","create\u002Fsv","dashboard\u002Fda","dashboard\u002Fde","dashboard\u002Fen","dashboard\u002Fno","dashboard\u002Fsv","enterprise-landingpage-2023\u002Fen","enterprise-landingpage\u002Fda","enterprise-landingpage\u002Fde","enterprise-landingpage\u002Fen","enterprise-landingpage\u002Fno","enterprise-landingpage\u002Fsv","financial\u002Fen","healthcare\u002Fen","industrial\u002Fen","investor\u002Fda","investor\u002Fen","investor\u002Fno","investor\u002Fsv","jobs\u002Fda","jobs\u002Fde","jobs\u002Fen","jobs\u002Fno","jobs\u002Fsv","kestrel\u002Fen","lifestyle\u002Fen","measure\u002Fda","measure\u002Fde","measure\u002Fen","measure\u002Fno","measure\u002Fsv","monitor\u002Fda","monitor\u002Fde","monitor\u002Fen","monitor\u002Fno","monitor\u002Fsv","ngo-landing-page-2023-simple\u002Fen","ngo-landingpage-2023\u002Fen","platform\u002Fda","platform\u002Fde","platform\u002Fen","platform\u002Fno","platform\u002Fsv","privacy-policy\u002Fen","property\u002Fen","report\u002Fda","report\u002Fde","report\u002Fen","report\u002Fno","report\u002Fsv","section-library\u002Fen","sme-landingpage\u002Fda","sme-landingpage\u002Fde","sme-landingpage\u002Fen","sme-landingpage\u002Fno","sme-landingpage\u002Fsv","terms-and-conditions\u002Fen","tmt\u002Fen","your-solution\u002Fda","your-solution\u002Fde","your-solution\u002Fen","your-solution\u002Fno","your-solution\u002Fsv",null,1793105304618]