host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://filebrowser.dandroid.app/robots.txt 200 7427 bytes 10 User-agent groups
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
User-agent: meta-externalagent
Disallow
  • /
Raw robots.txt 
# As a condition of accessing this website, you agree to abide by the following
# content signals:

# (a)  If a Content-Signal = yes, you may collect content for the corresponding
#      use.
# (b)  If a Content-Signal = no, you may not collect content for the
#      corresponding use.
# (c)  If the website operator does not include a Content-Signal for a
#      corresponding use, the website operator neither grants nor restricts
#      permission via Content-Signal with respect to the corresponding use.

# The content signals and their meanings are:

# search:   building a search index and providing search results (e.g., returning
#           hyperlinks and short excerpts from your website's contents). Search does not
#           include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
#           augmented generation, grounding, or other real-time taking of content for
#           generative AI search answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, user-scalable=no"
    />

    

    <title>
      filebrowser.dandroid.app
    </title>

    <meta name="robots" content="noindex,nofollow" />

    <link
      rel="icon"
      type="image/svg+xml"
      href="/static/img/icons/favicon.svg"
    />
    <link rel="shortcut icon" href="/static/img/icons/favicon.ico" />
    <link
      rel="apple-touch-icon"
      sizes="180x180"
      href="/static/img/icons/apple-touch-icon.png"
    />
    <meta name="apple-mobile-web-app-title" content="File Browser" />

    
    <link
      rel="manifest"
      id="manifestPlaceholder"
      crossorigin="use-credentials"
    />
    <meta
      name="theme-color"
      content="#2979ff"
    />

    
    <script>
      
      window.FileBrowser = {"AuthMethod":"json","BaseURL":"","CSS":false,"Color":"","DisableExternal":false,"DisableUsedPercentage":false,"EnableExec":false,"EnableThumbs":false,"HideLoginButton":false,"LoginPage":true,"LogoutPage":"/login","Name":"filebrowser.dandroid.app","NoAuth":false,"ReCaptcha":false,"ResizePreview":false,"Signup":false,"StaticURL":"/static","Theme":"dark","TusSettings":{"chunkSize":10485760,"retryCount":5},"Version":"2.63.2"};
      
      window.__prependStaticUrl = (url) => {
        return `${window.FileBrowser.StaticURL}/${url.replace(/^\/+/, "")}`;
      };
      var dynamicManifest = {
        name: window.FileBrowser.Name || "File Browser",
        short_name: window.FileBrowser.Name || "File Browser",
        icons: [
          {
            src: window.__prependStaticUrl("/img/icons/android-chrome-192x192.png"),
            sizes: "192x192",
            type: "image/png",
          },
          {
            src: window.__prependStaticUrl("/img/icons/android-chrome-512x512.png"),
            sizes: "512x512",
            type: "image/png",
          },
        ],
        start_url: window.location.origin + window.FileBrowser.BaseURL,
        display: "standalone",
        background_color: "#ffffff",
        theme_color: window.FileBrowser.Color || "#455a64",
      };

      const stringManifest = JSON.stringify(dynamicManifest);
      const blob = new Blob([stringManifest], { type: "application/json" });
      const manifestURL = URL.createObjectURL(blob);
      document
        .querySelector("#manifestPlaceholder")
        .setAttribute("href", manifestURL);
    </script>

    <style>
      #loading {
        position: fixed;
        top: 0;
        left: 0;
        width: 100%;
        height: 100%;
        background: #fff;
        z-index: 9999;
        transition: 0.1s ease opacity;
        -webkit-transition: 0.1s ease opacity;
      }

      #loading.done {
        opacity: 0;
      }

      #loading .spinner {
        width: 70px;
        text-align: center;
        position: fixed;
        top: 50%;
        left: 50%;
        transform: translate(-50%, -50%);
      }

      #loading .spinner > div {
        width: 18px;
        height: 18px;
        background-color: #333;
        border-radius: 100%;
        display: inline-block;
        animation: sk-bouncedelay 1.4s infinite ease-in-out both;
      }

      #loading .spinner .bounce1 {
        animation-delay: -0.32s;
      }

      #loading .spinner .bounce2 {
        animation-delay: -0.16s;
      }

      @keyframes sk-bouncedelay {
        0%,
        80%,
        100% {
          transform: scale(0);
        }
        40% {
          transform: scale(1);
        }
      }
    </style>
    <script type="module" crossorigin src="/static/assets/index-D9EC0xAu.js"></script>
    <link rel="modulepreload" crossorigin href="/static/assets/rolldown-runtime-Bhmf7a9N.js">
    <link rel="modulepreload" crossorigin href="/static/assets/dayjs-DtVdYEdL.js">
    <link rel="modulepreload" crossorigin href="/static/assets/i18n-DFXOUg7v.js">
    <link rel="stylesheet" crossorigin href="/static/assets/index-BtYCM-IR.css">
    <script type="module">import'data:text/javascript,if(!import.meta.resolve)throw Error("import.meta.resolve not supported")';import.meta.url;import("_").catch(()=>1);(async function*(){})().next();window.__vite_is_modern_browser=true</script>
    <script type="module">!function(){if(window.__vite_is_modern_browser)return;console.warn("vite: loading legacy chunks, syntax error above and the same error below should be ignored");var e=document.getElementById("vite-legacy-polyfill"),n=document.createElement("script");n.src=e.src,n.onload=function(){System.import(document.getElementById('vite-legacy-entry').getAttribute('data-src'))},document.body.appendChild(n)}();</script>
  </head>
  <body>
    <div id="app"></div>

    <div id="loading">
      <div class="spinner">
        <div class="bounce1"></div>
        <div class="bounce2"></div>
        <div class="bounce3"></div>
      </div>
    </div>


    
    <script nomodule>!function(){var e=document,t=e.createElement("script");if(!("noModule"in t)&&"onbeforeload"in t){var n=!1;e.addEventListener("beforeload",(function(e){if(e.target===t)n=!0;else if(!e.target.hasAttribute("nomodule")||!n)return;e.preventDefault()}),!0),t.type="module",t.src=".",e.head.appendChild(t),t.remove()}}();</script>
    <script nomodule crossorigin id="vite-legacy-polyfill" src="/static/assets/polyfills-legacy-D_7TL9qj.js"></script>
    <script nomodule crossorigin id="vite-legacy-entry" data-src="/static/assets/index-legacy-QjigCFw-.js">System.import(document.getElementById('vite-legacy-entry').getAttribute('data-src'))</script>
  </body>
</html>
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Ffilebrowser.dandroid.app 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Ffilebrowser.dandroid.app'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view