robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools

Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector

https://learn.github.com/robots.txt 200 2838 bytes 0 User-agent groups

Raw robots.txt

<!doctype html>
<html lang="en" data-color-mode="dark">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta name="content-engine-url" content="https://cseafdprod-dxcabeg6ega6fggz.b01.azurefd.net" />
    <meta name="support-url" content="https://support.github.com" />
    <link rel="shortcut icon" href="/favicon.svg" type="image/svg+xml" />

    <title>GitHub Learn</title>
    <meta name="description" content="GitHub Learn" data-react-helmet="true" />

    <meta property="og:title" content="GitHub Learn: Your Personalized Learning Experience" />
    <meta
      property="og:description"
      content="GitHub Learn is the all-in-one learning experience platform that unifies GitHub’s official learning and enablement resources into personalized journeys. Whether you're pursuing certification or want to learn about one of our new features, GitHub Learn helps you set goals, track progress, and build the skills that matter — all from one trusted source."
    />
    <meta property="og:type" content="website" />
    <meta property="og:image" content="https://github.githubassets.com/assets/github-mark-57519b92ca4e.png" />

    <meta property="twitter:site" content="github" />
    <meta property="twitter:site:id" content="13334762" />
    <meta property="twitter:creator" content="github" />
    <meta property="twitter:creator:id" content="13334762" />
    <meta property="twitter:card" content="summary_large_image" />
    <meta property="twitter:title" content="GitHub" />
    <meta
      property="twitter:description"
      content="GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects."
    />
    <meta property="twitter:image" content="https://github.githubassets.com/assets/github-logo-55c5b9a1fe52.png" />
    <script type="module" crossorigin src="/assets/index-BeIS62MC.js"></script>
    <link rel="modulepreload" crossorigin href="/assets/chunk-QTnfLwEv.js">
    <link rel="modulepreload" crossorigin href="/assets/preload-helper-zJ_50EbN.js">
    <link rel="modulepreload" crossorigin href="/assets/useTranslation-BjERsjip.js">
    <link rel="modulepreload" crossorigin href="/assets/useDevOnlyEffect-AGL25hoA.js">
    <link rel="modulepreload" crossorigin href="/assets/index.esm-xS5KEw19.js">
    <link rel="modulepreload" crossorigin href="/assets/react-dom-DTOgVJK5.js">
    <link rel="modulepreload" crossorigin href="/assets/Dialog-ECyhML78.js">
    <link rel="stylesheet" crossorigin href="/assets/useDevOnlyEffect-ZB9z74Fn.css">
    <link rel="stylesheet" crossorigin href="/assets/Dialog-BuAOKa0O.css">
    <link rel="stylesheet" crossorigin href="/assets/index-Dt4Lq-jO.css">
  </head>
  <body>
    <div id="root"></div>
  </body>
</html>

How to use robots.txt parser

1

Paste your input

Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.
2

Click "Inspect"

host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.
3

Open the API tab

Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent

/api/v1/http/robots?q=https%3A%2F%2Flearn.github.com

curl -s '/api/v1/http/robots?q=https%3A%2F%2Flearn.github.com'

Embed this tool

<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?

Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.

Where does the data come from?

Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).

How fresh are the results?

Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.

Can I run this from the command line?

Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.

Can I monitor results over time?

Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.

host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors

✓Schedule any tool — every 1, 5, 15, 60 min
✓Diff against last run, alert on change
✓Webhook + email + Slack + PagerDuty + OpsGenie
✓Bulk CSV upload, 1,000 inputs per job
✓Export results as CSV / NDJSON / Excel
✓90-day history, comparison view