host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://host.com/robots.txt 403 5279 bytes 0 User-agent groups
Raw robots.txt 
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><meta http-equiv="content-security-policy" content="default-src &#39;none&#39;; script-src &#39;nonce-epNptvzA3rY0jFGailDwSR&#39; &#39;unsafe-eval&#39; https://challenges.cloudflare.com; script-src-attr &#39;none&#39;; style-src &#39;unsafe-inline&#39;; img-src &#39;self&#39; https://challenges.cloudflare.com; connect-src &#39;self&#39; https://challenges.cloudflare.com; frame-src &#39;self&#39; https://challenges.cloudflare.com blob:; child-src &#39;self&#39; https://challenges.cloudflare.com blob:; worker-src blob:; form-action http: https:; base-uri &#39;self&#39;"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;padding-left:1.5rem;max-width:60rem}@media (width <= 720px){.main-content{margin-top:4rem}}#challenge-error-text{background-image:url("data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+");background-repeat:no-repeat;background-size:contain;padding-left:34px}</style><meta http-equiv="refresh" content="360"></head><body><div class="main-wrapper" role="main"><div class="main-content"><noscript><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></noscript></div></div><script nonce="epNptvzA3rY0jFGailDwSR">(function(){window._cf_chl_opt = {cFPWv: 'g',cH: 's23BsAzP8LJ77cEUWnfDLaawgRMEUgNkVEhlRR9fWIA-1778300411-1.2.1.1-43.cfDYLJmKkx17ifxJbCvOM9_ys16tLO.072Ki6nzEJZ6IYqsx0yIHGVp.B7BMx',cITimeS: '1778300411',cN: 'epNptvzA3rY0jFGailDwSR',cRay: '9f8de9066b061c96',cTplB: '0',cTplC:0,cTplO:0,cTplV:5,cType: 'managed',cUPMDTk:"/robots.txt?__cf_chl_tk=L4MMc8Omi0ctnbyBthWPbUMkM1aV5Ytgd35nV7KM6c8-1778300411-1.0.1.1-ZAZWREOyJdJ6LRuyr2N4JETBg7eHgAiHygpe4Xsk9YY",cvId: '3',cZone: 'host.com',fa:"/robots.txt?__cf_chl_f_tk=L4MMc8Omi0ctnbyBthWPbUMkM1aV5Ytgd35nV7KM6c8-1778300411-1.0.1.1-ZAZWREOyJdJ6LRuyr2N4JETBg7eHgAiHygpe4Xsk9YY",md: '0YI8WEkDyNv4L02ZFi2CdPGKhafUFRrDmozo7yvuJD8-1778300411-1.2.1.1-x2v3q.VJMvizhKRk5QBn0EYkXUWKvgET72zMcD1OdBAeCDqTwz_ZH8y90aN_ia.5SC3...j7VEXRj7NHWHXNkCWAgcstX1Vfe6Qwr2U79TGDafSOpo3CPN4Cy.KHkU70HjmyIwJNxsoFIdJnY2s1E9VtdUAj4Ilcr6enbuGabUvXBEBTwszGinR3faeRRBTN839sm8WICFUbXs28Jvm8Me.X5BrTyV4u7_qTRLYQgEcxiiT9y0EMMuu1jpaMh0uFMTPTfZgQW.jwajgPFAZv7GIuO5GnmM4jgeANDGIVxI4dEclbj6TmiS3QkP2zgw03X4vuDmFj7sTBOUg7VrKzh1A_AKfNE3bJbkioO67hIzs7ed7O3qzI.8iLmaA7Eh1umU0Wt46RqWagDfH3ENvZ1XGOb5sTAnaa7VSWXuP.lEYLKP4Ts7EssY53pCWTiA20AAMKZsOpArFD9cKv6d48uq8YOq_gshtQp0pU7.eAqUIBLOyJAGH.9Go.yVQOYvJR_0WsTWB2ztU.hVMlhhjSOYbuFEnKvrHKd70hfo8zxBpmMvcxHrfPZ0PcIn_k_IpGIRkacJu2jFMzA.1.gHxbZ7.ChoHdOTZu59wFAmjxJtNkbaEixHA7OIEY1dUMKW0hGVn5ejzzV_Aq1bIelBA2ithhYufO4RtVqsPxh8jjTLyH5QqA7RdX7QXFV3jDCmGWa5dZUR5JNpmjPHfINCQUyGzRzzwWx_PyUTHBC1Ti0XrDw7cteYB9u4jovZFPmOLoYyzoK4Per_stjRLPvyuWJSlHmMiePHBR3mrWhcpS_Zus0DrUyCzdNtZHFBBx.3htYfW3aeMeElKawRVb9_QVjhKRy5_TdkWPOG6ZeT2I_e0',mdrd: 'kb52CrZdSAZ_dNSrwr0sQUPUrpQol1syrvVKXPnrf2Y-1778300411-1.2.1.1-8DcqTWlp2nrejUHLLPOI6pxhoVErPRPFizk91rPwZGtBSbwSWv8ZAMuJFr_dyC9xJ3L7kn8b_3gG7p5Rghf6OrwhRfyiJ6s0LhnKoYRpRc7HV058ipeSq9PRFxJCwYor_ymSu5DSRacq5mEdaRBTVrA6vvHXtroAiix5vLxSzkOlinG9n9WzTpDpLr7bpumGjLLrIH9wl7jrgPhF0ijoZBoHiXIJwzcF_Ft6rMnTvD8bLKr4zPVhbu0LE5oJ2GQfNFtXVzxPPkKp87ItnEktBon5UG4gVPMvg46WXJtBTeR_ZLJ6f.rBRYBrXbS8CedK',};var a = document.createElement('script');a.nonce = 'epNptvzA3rY0jFGailDwSR';a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=9f8de9066b061c96';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null,"/robots.txt?__cf_chl_rt_tk=L4MMc8Omi0ctnbyBthWPbUMkM1aV5Ytgd35nV7KM6c8-1778300411-1.0.1.1-ZAZWREOyJdJ6LRuyr2N4JETBg7eHgAiHygpe4Xsk9YY"+ window._cf_chl_opt.cOgUHash);a.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(a);}());</script></body></html>
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fhost.com 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fhost.com'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view