host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://host.com/robots.txt 403 5343 bytes 0 User-agent groups
Raw robots.txt 
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><meta http-equiv="content-security-policy" content="default-src &#39;none&#39;; script-src &#39;nonce-n4zAA3Vk67aGdzH8ZnZlFN&#39; &#39;unsafe-eval&#39; https://challenges.cloudflare.com; script-src-attr &#39;none&#39;; style-src &#39;unsafe-inline&#39;; img-src &#39;self&#39; https://challenges.cloudflare.com; connect-src &#39;self&#39; https://challenges.cloudflare.com; frame-src &#39;self&#39; https://challenges.cloudflare.com blob:; child-src &#39;self&#39; https://challenges.cloudflare.com blob:; worker-src blob:; form-action http: https:; base-uri &#39;self&#39;"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;padding-left:1.5rem;max-width:60rem}@media (width <= 720px){.main-content{margin-top:4rem}}#challenge-error-text{background-image:url("data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+");background-repeat:no-repeat;background-size:contain;padding-left:34px}</style><meta http-equiv="refresh" content="360"></head><body><div class="main-wrapper" role="main"><div class="main-content"><noscript><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></noscript></div></div><script nonce="n4zAA3Vk67aGdzH8ZnZlFN">(function(){window._cf_chl_opt = {cFPWv: 'g',cH: 'eVwv9rC71sQ3HpXmxFcWSYUccbwechv.wPC5.Js4K_Q-1782197244-1.2.1.1-WqwpSQexbTMdKOXmc5ZOM5nxUMESbjI7GpOy6UKwHdIS1vgLYSyeoaP_bzlEe7.j',cITimeS: '1782197244',cN: 'n4zAA3Vk67aGdzH8ZnZlFN',cRay: 'a1018a8acade7869',cTplB: '0',cTplC:0,cTplO:0,cTplV:5,cType: 'managed',cUPMDTk:"/robots.txt?__cf_chl_tk=8.2IEmkdBnnTn.PM2uwKmwg6GGUBj6A4fbghYsSO1Qc-1782197244-1.0.1.1-gaJFt.Sh2H6VZA2srsXk74YcMfzkTn27XsorBKSoH8E",cvId: '3',cZone: 'host.com',fa:"/robots.txt?__cf_chl_f_tk=8.2IEmkdBnnTn.PM2uwKmwg6GGUBj6A4fbghYsSO1Qc-1782197244-1.0.1.1-gaJFt.Sh2H6VZA2srsXk74YcMfzkTn27XsorBKSoH8E",md: '_3SMeLjLpDPO59cUEryuf22E5Mugw0yYqps0BW6IgLk-1782197244-1.2.1.1-wrUnfLtJHZP1EFQBW5d16t.ME6KPMG8VB5y_0p4n4YKiMCxAb89pmAcFMfbPm30XhkyMdCx8b4q_jRNJ.34cNYN9BEdXldVPxxqDpdm01HfBHDdgL.W4dpjc.VFZwtiENNgURrW_K5ZZBpbeKALsFTns5oxLxhAeT40boQdVE7khQRjFZ4_NGqN2sk5mgC0GJhKcLsjofF6EKlahKJ4WsvkkSBpdG1YP3e85Qcc.AVYm_Vb858VdZ9B3Jw0hom4Kl0vDR7V3koYj2SOQ1hjp3etqtMsFbhVcfm3EWhwSbznVX5VhN.HcmnrHlBP7lw7Xyki3nSPhiC31lpIk2vvwcV0RQvyYlB9kmeaPvOFo1PerHck5hO1EYVTlh9rRZ1KhxNtLHpxQx8DtUCqZosaKEjzy_RVlXH9luaK.I5_JFyhk5plZBexdo36jHNqiXfRrWbLVB30CBcMXgr8HZvYWdEFOhFBJjEyR.AJ5w1qDSFPFibEvT0Wj5SuclUx72Mv0OdRrAoIpsujNpHCImmuZ1O_ioLl1olEXsHZdcQDHBk_8Myqyfj.WUeQ_qeDPxRQChACMaMa_op07N6LeOgVQMZqBMoDtu8YnGV.puOf9_MsYh_Ubrc23XgwDHp0dWq5..qZF5QrUF8yDz4ws0opFjusnQU4RzG9tcLCnQHN8Y726eF_Mtni1b4KmiUvezalHZG0TiZWv9Rj8Hcbb6sK08FgX0O.ugPitah7SdpxsnwSo2p69Yguy4BSuDNZXQhUxeIRR4Lry.VjDHMSk1B0fjgGdLYlrv2bOzt179GFXtk1Q9jIiVSXRY56.gnxLfEZS71FQMNuq.KePubYQ4h9oYqjsuaAHRgZfKMIGNhNJHRE6VxVhUhjaUFONBjqWpC7qvLy0X_zhTdX64REhuuxtIVNiPo2fxW3fj12L7Kx7huUNG2exdV5fduTcml0v5Lc4qaUMzXJdl1yYd_4ZV3_bC0yu93ZT7doQcbwp3BuF5K4',mdrd: 'w_HCYdy.WtlAL_0qHmvnaFcmq09IwSFOlRpgB.aGmdA-1782197244-1.2.1.1-yQaazFKtFX_Gq_1jdUzNnUEjg4F40n7BRSexcsetogudHthjl70AzPqo69MuQO6S9Uza.0O2ZSiaGHFCE9CT_EmDvTxWGofHyHUqyqVGLXzJ6.YI1mqx.lbZQPLaqQ.0u2KPt1P_hK.MkSrevQ05uKZzykxVodJg8VVKLSmm2dfb65V5OdEvJqb6PWiGL2xZHKtqmVUNjk.WA7LG7T2mEw3LSO2w_HlVbRMIkB5eNsDlQ1Fx_VP0xGTtBF.J7Ptq',};var a = document.createElement('script');a.nonce = 'n4zAA3Vk67aGdzH8ZnZlFN';a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=a1018a8acade7869';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null,"/robots.txt?__cf_chl_rt_tk=8.2IEmkdBnnTn.PM2uwKmwg6GGUBj6A4fbghYsSO1Qc-1782197244-1.0.1.1-gaJFt.Sh2H6VZA2srsXk74YcMfzkTn27XsorBKSoH8E"+ window._cf_chl_opt.cOgUHash);a.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(a);}());</script></body></html>
Sponsored host.tools Pro
Bulk lookups · monitors · webhooks · 10,000 API calls/day

Upgrade to Pro for $19/mo. Cancel anytime. Works with the same API you already use.
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fhost.com 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fhost.com'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view