robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools

Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector

https://portal.dev.wearecurie.ai/robots.txt 404 5454 bytes 0 User-agent groups

Raw robots.txt

<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><title>Curie</title><meta name="viewport" content="minimum-scale=1, initial-scale=1, width=device-width, user-scalable=no"/><link rel="shortcut icon" href="/favicon.png"/><link crossorigin="anonymous" href="https://EGELT26TVA-dsn.algolia.net" rel="preconnect"/><meta name="next-head-count" content="5"/><script data-mantine-script="true">try {
  var _colorScheme = window.localStorage.getItem("mantine-color-scheme-value");
  var colorScheme = _colorScheme === "light" || _colorScheme === "dark" || _colorScheme === "auto" ? _colorScheme : "light";
  var computedColorScheme = colorScheme !== "auto" ? colorScheme : window.matchMedia("(prefers-color-scheme: dark)").matches ? "dark" : "light";
  document.documentElement.setAttribute("data-mantine-color-scheme", computedColorScheme);
} catch (e) {}
</script><link rel="preload" href="/_next/static/media/32f21afdec8be873-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/_next/static/css/69c5023b30385812.css" as="style"/><link rel="stylesheet" href="/_next/static/css/69c5023b30385812.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" nomodule="" src="/_next/static/chunks/polyfills-78c92fac7aa8fdd8.js"></script><script src="/_next/static/chunks/webpack-ad6e6414a43ec594.js" defer=""></script><script src="/_next/static/chunks/framework-0e8d27528ba61906.js" defer=""></script><script src="/_next/static/chunks/main-286ad989d69d424c.js" defer=""></script><script src="/_next/static/chunks/pages/_app-34d88661a215ed7c.js" defer=""></script><script src="/_next/static/chunks/pages/404-86ba92a99d29b1ea.js" defer=""></script><script src="/_next/static/d9xh1OXmPUDV4tN0IGqMp/_buildManifest.js" defer=""></script><script src="/_next/static/d9xh1OXmPUDV4tN0IGqMp/_ssgManifest.js" defer=""></script></head><body><div id="__next"><style data-mantine-styles="classes">@media (max-width: 35.99375em) {.mantine-visible-from-xs {display: none !important;}}@media (min-width: 36em) {.mantine-hidden-from-xs {display: none !important;}}@media (max-width: 47.99375em) {.mantine-visible-from-sm {display: none !important;}}@media (min-width: 48em) {.mantine-hidden-from-sm {display: none !important;}}@media (max-width: 61.99375em) {.mantine-visible-from-md {display: none !important;}}@media (min-width: 62em) {.mantine-hidden-from-md {display: none !important;}}@media (max-width: 74.99375em) {.mantine-visible-from-lg {display: none !important;}}@media (min-width: 75em) {.mantine-hidden-from-lg {display: none !important;}}@media (max-width: 87.99375em) {.mantine-visible-from-xl {display: none !important;}}@media (min-width: 88em) {.mantine-hidden-from-xl {display: none !important;}}</style><style data-mantine-styles="inline">:root{--app-shell-navbar-width:0rem;--app-shell-navbar-offset:0rem;--app-shell-header-height:0rem;--app-shell-header-offset:0rem;--app-shell-padding:0px;}@media(max-width: 36em){:root{--app-shell-navbar-width:100%;--app-shell-navbar-offset:0px;}}</style><div style="--app-shell-transition-duration:500ms;--app-shell-transition-timing-function:ease" class="m_89ab340 mantine-AppShell-root" data-resizing="true"><main class="m_8983817 mantine-AppShell-main"><div style="padding:var(--mantine-spacing-xl)" class="centered-container m_7485cace mantine-Container-root"><div style="--stack-gap:var(--mantine-spacing-md);--stack-align:center;--stack-justify:flex-start" class="m_6d731127 mantine-Stack-root"><svg xmlns="http://www.w3.org/2000/svg" width="15rem" height="15rem" viewBox="0 0 24 24" fill="none" stroke="var(--mantine-color-red-3)" stroke-width="1" stroke-linecap="round" stroke-linejoin="round" class="tabler-icon tabler-icon-error-404 " style="stroke-width:1px"><path d="M3 7v4a1 1 0 0 0 1 1h3"></path><path d="M7 7v10"></path><path d="M10 8v8a1 1 0 0 0 1 1h2a1 1 0 0 0 1 -1v-8a1 1 0 0 0 -1 -1h-2a1 1 0 0 0 -1 1z"></path><path d="M17 7v4a1 1 0 0 0 1 1h3"></path><path d="M21 7v10"></path></svg><h4 style="--title-fw:var(--mantine-h4-font-weight);--title-lh:var(--mantine-h4-line-height);--title-fz:var(--mantine-h4-font-size);color:var(--mantine-color-dimmed);text-align:center" class="m_8a5d1357 mantine-Title-root" data-order="4">Page not found</h4><p style="color:var(--mantine-color-dimmed)" class="mantine-focus-auto m_b6d8b162 mantine-Text-root">We have searched high and low but could not find what you are looking for :/</p><a style="text-decoration:none" href="/documents/search"><button style="--button-height:var(--button-height-sm);--button-padding-x:var(--button-padding-x-sm);--button-fz:var(--mantine-font-size-sm);--button-bg:transparent;--button-hover:var(--mantine-color-blue-outline-hover);--button-color:var(--mantine-color-blue-outline);--button-bd:calc(0.0625rem * var(--mantine-scale)) solid var(--mantine-color-blue-outline)" class="mantine-focus-auto mantine-active m_77c9d27d mantine-Button-root m_87cf2631 mantine-UnstyledButton-root" data-variant="outline" data-size="sm" data-block="true" type="button"><span class="m_80f1301b mantine-Button-inner"><span class="m_811560b9 mantine-Button-label">Go back Home</span></span></button></a></div></div></main></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{}},"page":"/404","query":{},"buildId":"d9xh1OXmPUDV4tN0IGqMp","nextExport":true,"autoExport":true,"isFallback":false,"scriptLoader":[]}</script></body></html>

How to use robots.txt parser

1

Paste your input

Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.
2

Click "Inspect"

host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.
3

Open the API tab

Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent

/api/v1/http/robots?q=https%3A%2F%2Fportal.dev.wearecurie.ai

curl -s '/api/v1/http/robots?q=https%3A%2F%2Fportal.dev.wearecurie.ai'

Embed this tool

<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?

Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.

Where does the data come from?

Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).

How fresh are the results?

Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.

Can I run this from the command line?

Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.

Can I monitor results over time?

Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.

host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors

✓Schedule any tool — every 1, 5, 15, 60 min
✓Diff against last run, alert on change
✓Webhook + email + Slack + PagerDuty + OpsGenie
✓Bulk CSV upload, 1,000 inputs per job
✓Export results as CSV / NDJSON / Excel
✓90-day history, comparison view