host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://host.com/robots.txt 403 5332 bytes 0 User-agent groups
Raw robots.txt 
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><meta http-equiv="content-security-policy" content="default-src &#39;none&#39;; script-src &#39;nonce-m086fZu8uRThQQS446b0tg&#39; &#39;unsafe-eval&#39; https://challenges.cloudflare.com; script-src-attr &#39;none&#39;; style-src &#39;unsafe-inline&#39;; img-src &#39;self&#39; https://challenges.cloudflare.com; connect-src &#39;self&#39; https://challenges.cloudflare.com; frame-src &#39;self&#39; https://challenges.cloudflare.com blob:; child-src &#39;self&#39; https://challenges.cloudflare.com blob:; worker-src blob:; form-action http: https:; base-uri &#39;self&#39;"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;padding-left:1.5rem;max-width:60rem}@media (width <= 720px){.main-content{margin-top:4rem}}#challenge-error-text{background-image:url("data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+");background-repeat:no-repeat;background-size:contain;padding-left:34px}</style><meta http-equiv="refresh" content="360"></head><body><div class="main-wrapper" role="main"><div class="main-content"><noscript><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></noscript></div></div><script nonce="m086fZu8uRThQQS446b0tg">(function(){window._cf_chl_opt = {cFPWv: 'g',cH: 'eMg2aYfm6qQH9BL0r_W4ipukzz9wsXul0psm2ZTOXBo-1793404243-1.2.1.1-44BguNkg8fI4peyqZzVtcQRIvewr4qkXwCOZXZ.qKzl3igk3J2MPjXoRy2EyD72m',cITimeS: '1793404243',cN: 'm086fZu8uRThQQS446b0tg',cRay: 'a52e52ebab1490cc',cTplC:0,cTplO:0,cTplV:5,cType: 'managed',cUPMDTk:"/robots.txt?__cf_chl_tk=bQcZObWxqX82Y3Biz6qHhb3qhPn5RDy..CL2wZOHbvg-1793404243-1.0.1.1-1RykL1KVG8I0XeznJaJG9MsZyJVdK.GxWCOc_SZorYo",cvId: '3',cZone: 'host.com',fa:"/robots.txt?__cf_chl_f_tk=bQcZObWxqX82Y3Biz6qHhb3qhPn5RDy..CL2wZOHbvg-1793404243-1.0.1.1-1RykL1KVG8I0XeznJaJG9MsZyJVdK.GxWCOc_SZorYo",md: 'BttDGlei.GDm22g5nWDjTmLURkwdvz33JpkEwdZY4Yo-1793404243-1.2.1.1-mcrM.SWLp3WPlTiZtBEgyu4y0AOYOkfwCdwA7gSstOn_IeC9yyophoQRPMOUsyu56gFv7sBft43L.IfCkG9Lhyh1V5aEoKcMN_nk1ZwhmstwIRE0uM6CW11yWYzyI8p9Kx_DhDa_EpTX_x7_b2nl23AoPTn8OtyW3m8jeRCFopEkrhCxccBdlDyV_uJqwUutUMU0Khj.V7anET1q_nXrGHKBfJ2qQLgg6t5OXEvJUhDlbBRqQ8O7VjXqTwiEL4VLSSxLWpPFSdrEKp3Zm2WMkc2KvenwZgaHGxMTMVna7SpiABl9BU_AQbCjLwlRkHF8vMgaAKXbCMvnS5nQkX_6nolXSZekBZXtMNIbNHUaPUYYcKD80FIEUznxJ1n0cLv2NlhvJHDft2sYa._9fzliOVauWYsLAnTmN7GSBNZgme2.9ygARTT92lJ305Mwhg9no5PFQQ7Rj9E0GCIlkYqYgB0lozbl.cKW.PdPYTrD35vqg.lO53BhPJ9iU.lTpqMxJJ7M0LaWzaE2n57SBna2YJseNXySEDqNjghuamwPzZTW0_bebDqDFd94.C_6xO387obR8wUHQqRckNAok3G2nhlDtBIV3t7YWNNh4D4ZNrLP.wxlSH2OGd4ODT.UfuDyRtckHx_70z2OoNXLTPXdM36lkhe6vtrxs2RgHz21dKT.xXmT1UW68kO2Uz5t_1I.2tgshw77GBuGb2kDo8q.iOyLqwDOjBdHXYlJ35hiZgGJ5Kqsho0o2joBO3aF7m5hhszdBPNbsxhktM8QDdbrk3eVSzRfLHATZ6PwUq0aNgDQxd9EEYo8Z6htTB8X3RAS7Zdmc_PBlaiKX3grLLsxLNH.6WEjlw3x9an2z5vuPDDqcRpvjFIekXRwOG_KBn2OpaU94a71yxFyCUGyj8u5VAZRfdDP1LArtEDKHgOXtfsTMt4V0L.c_XQkWPRMgNBdinv8NGPbpiUVAMiAYPmkgkn9icVJuTJG8slvLbwbPlM',mdrd: 'GMQleBuigV3Hhn5rC7JtuHrGmJhoIfRomrebmYeAuMc-1793404243-1.2.1.1-K1agDTfm4EQIEe1iDd4240EHmlzjr55jVaMM2Zi1k0jOnNQHYdjMXzgCewc12DyDtJ1fLFMdzS1UoWzydSq2hWyCMeurvq9h9AYxlSqXjaVxCcv7Se5l67K4KVrUE4SCIwYcv3Wn0PWM442l8jOca6lVp_eIb_PJAJtVQyJoVUDMIaaybxCNTmy_4.00IS_ds4nlM4_yubIuJcQOQSLWJJld9zyt58ZQjo42NYVcdfkwdECv.i3.YTY49B3AM2r9',};var a = document.createElement('script');a.nonce = 'm086fZu8uRThQQS446b0tg';a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=a52e52ebab1490cc';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null,"/robots.txt?__cf_chl_rt_tk=bQcZObWxqX82Y3Biz6qHhb3qhPn5RDy..CL2wZOHbvg-1793404243-1.0.1.1-1RykL1KVG8I0XeznJaJG9MsZyJVdK.GxWCOc_SZorYo"+ window._cf_chl_opt.cOgUHash);a.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(a);}());</script></body></html>
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fhost.com 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fhost.com'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view