host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://host.com/robots.txt 403 5343 bytes 0 User-agent groups
Raw robots.txt 
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><meta http-equiv="content-security-policy" content="default-src &#39;none&#39;; script-src &#39;nonce-l81RaRcjLE5ipbcjrOOCrC&#39; &#39;unsafe-eval&#39; https://challenges.cloudflare.com; script-src-attr &#39;none&#39;; style-src &#39;unsafe-inline&#39;; img-src &#39;self&#39; https://challenges.cloudflare.com; connect-src &#39;self&#39; https://challenges.cloudflare.com; frame-src &#39;self&#39; https://challenges.cloudflare.com blob:; child-src &#39;self&#39; https://challenges.cloudflare.com blob:; worker-src blob:; form-action http: https:; base-uri &#39;self&#39;"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;padding-left:1.5rem;max-width:60rem}@media (width <= 720px){.main-content{margin-top:4rem}}#challenge-error-text{background-image:url("data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+");background-repeat:no-repeat;background-size:contain;padding-left:34px}</style><meta http-equiv="refresh" content="360"></head><body><div class="main-wrapper" role="main"><div class="main-content"><noscript><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></noscript></div></div><script nonce="l81RaRcjLE5ipbcjrOOCrC">(function(){window._cf_chl_opt = {cFPWv: 'g',cH: 'Ez3EJUfxZQGUyMHrZf_5xTiUMS289qIpjMIXI3XuD4s-1789502063-1.2.1.1-R_pv9zNzTYLxjh50hkr.j_RNNJ8al0a1JoIqcS0HFnQT8mFUTreqwecprV5k5pia',cITimeS: '1789502063',cN: 'l81RaRcjLE5ipbcjrOOCrC',cRay: 'a3ba2ed5f9053545',cTplB: '0',cTplC:0,cTplO:0,cTplV:5,cType: 'managed',cUPMDTk:"/robots.txt?__cf_chl_tk=oJhQKbtnhst4Vz5WvcdUOJMKeuUSYla4tWymm_ZSdic-1789502063-1.0.1.1-vpc6jv9z6xNXs.hz0Locx4sieGXrrov7j6I7H4Gydrw",cvId: '3',cZone: 'host.com',fa:"/robots.txt?__cf_chl_f_tk=oJhQKbtnhst4Vz5WvcdUOJMKeuUSYla4tWymm_ZSdic-1789502063-1.0.1.1-vpc6jv9z6xNXs.hz0Locx4sieGXrrov7j6I7H4Gydrw",md: 'OR_7rrcVm5xJjMLaae24Bb972KEiNoZheIrOp_WHPbI-1789502063-1.2.1.1-6dtXTqg9T7Tbh5eZ3xH5Y2bQ4AKX08UT3Bk23xlBm_VWIPJbFn8KnW8OyR0XYW5pxpLVt6BG96dG61oEOiw_eW31aIBFrgztnr2P323aVKw9bcprqcIr75vAzwIzQ4JgWYAGxMYuvuh..RGt_nxcr6gFdozBcHSF5ebMRaZr7HRpZdbknAsGr5KTsMA4WSYsoj.zXume7xYzQNzwIY0o9mLIVzQHi1acxOfF1HI2xlDzzoJ4I9lx9Kkx4vlOUh3eihCkLwlZleVDNC.cxgNFIJznKnwUp_DUyaKOejJZSopGOdIouyyfmXZV09EEAURC4gGvcpB8NFJMU8yZxyqs9Z8mtI_vPwDAEHj.84lyKP3B4WNV0VbXfsQA7wRZTgSh8iJhFdn_AImLLZzgByzlylehZpgZu4qdk53RsLqqHmPuBtw6Ytmcb2T4qMH7cbVsbL2WuuiVDt9EL6WQTdZLsmA8Dz6p.wR8hXIX9QHlN7nnut700BmOdjOyMw4CDWs24i_dIZvY123aGUKE4cRp08icxV7KrECIE4XQoqk5AhCpUsvtyr9HpwjWII.XYQ7UEUqvRclVOfUwFvS5CO4DxeTFnH2v7.xENTZhwFChDT27fSJ6y2aNXDgkJ_MeJml0.c9jUhWq1P05w9WbBxmMNUDUNYi4c3XebincP0JejIeqr9tsaEHS0AKB3HMN.3UCE_x7cpRzyk0iEmTMlnr_kewlVXRtijbyZynzQf4rfwgQBaadEGzA6nNgW1DWH0u.2EkXtNwC00D4_ceKq5YSyKSTl3IOgz.dFA4rNheUG9xsV3ddOy0dWJ1l._C7yopziDjj674cTn54Y3xE1VPVy40njFoH3d8OwMxUE9zAyKFIiplGLcbtg3_SditO5VKrD2lL8gN4roeLndXeREOuoESnLk7_vP8P0nA.P1yUPluq7Y9t4r.GZSsn1Wn6shGOT5yDc7C8WovpxiPpljA7U1rPHIf1AHljsXR9SZpMjHg',mdrd: 'FgErRQ0zLxY.LieLIlYwfVFXWuB5A1CKogCeiozYF0s-1789502063-1.2.1.1-o2tAs7HpihQFzWIMyi5esiKrVZafV4Fof7vv6bZ519PxKv8XdslmlAzYAnQOaKYnOSZ8gj.dCyldyHGjj2FlA6q_FZm.A87ohYN5b7qV9UT13p5gc.3y1hkc4Z8YijCQ1wdFTHi0yftKAURcb28xV1MvCJHjgzHgMf4F.MeXTjwuI0qyZIJS5eZAuH9zG1g7nSQNNzLH3j44QcL0fuwCP590j_PyZaiKab04a5vYe3xbBBLCX2Jj2atmri2w6SLU',};var a = document.createElement('script');a.nonce = 'l81RaRcjLE5ipbcjrOOCrC';a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=a3ba2ed5f9053545';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null,"/robots.txt?__cf_chl_rt_tk=oJhQKbtnhst4Vz5WvcdUOJMKeuUSYla4tWymm_ZSdic-1789502063-1.0.1.1-vpc6jv9z6xNXs.hz0Locx4sieGXrrov7j6I7H4Gydrw"+ window._cf_chl_opt.cOgUHash);a.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(a);}());</script></body></html>
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fhost.com 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fhost.com'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view