robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools

Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector

https://api.security.github.com/robots.txt 200 5139 bytes 0 User-agent groups

Raw robots.txt


<style>
pre { font-family: "Courier New", Courier, monospace }
</style>

<pre>
....................................................................................................
....................................................................................................
...........................................................'........................................
.........................................................'x0:.......................................
....................................................... 'kNO,.......................................
........................................................oNNO;.......................................
........................................................;KWXk:......................................
.........................................................:KWXOc.....................................
..........................................................:kKXXO:...................................
............................................................';lOc...................................
.............................................................,dx' ..................................
........................................ .... .............ckOx;....................................
......................................';codol:,......     .ck:......................................
...................................,ldO0K000K00kd;..       ;KKc. ...................................
................................,lx0000000OkO00OOOko.      .k0;    .................................
..............................'oO0000000000xd00xl;'..       ';.      ...............................
.........................    'x000000000000doO000l..       .:o.       ..............................
........................;cllcd000000000000kkO000Kxl'      .oXNo.      ..............................
..................... .dNWWWXK000000000000000000KOx;     .xNNx'       .,;;;,........................
..................... .dWWWNK0000000000000000000KOk;    ,OWXx'         .;;;,........................
.......................:XWXkxkO00000000000000000KOd,   .xNWk.          .';;'........................
...................,:;;cONk:;:ccloddxkkOO00000000o,.   .'o0Ko'..;,.     .,,.    ....................
...................cO000KKd:;;;;;;;;;;::cclllooo:.        .cxo;xNO.     ...     ....................
....................,lxO000Oxdolcc:;;;;;;;;;;,'.            .  oNk.            .....................
.......................,:ldxO0000Okkxddoolllc:.             .'oKKc.         ........................
........................  'ddccccodxkO0000000x,            ,xXNKl.   ... ...........................
......... ..............'.,kKool. ...'',;::c;.           .'kWWKl.    ..  ...........................
.........'........''',,,,;,cOOx0k;.         .,clo;.      .xNWWO'    ...  ...........................
....................'''.....'dOkKXOo:,'',:okKNWWXc..     cKXKWX:   ... .............................
...........''......   ........;dOKXNWNXXNWWWWWWXo.     .;kOooOKl. ..  ..............................
...........  ...................;lkKNWWWWWWWWWXKx'  ...'::::;;;..   ................................
...................................,d00KXNWWWWWWNc    ..';;;;,.  ...................................
....................................':;cxdccclloo'       ...........................................
..................................... .:xxlc:,'..    .....  ........................................
......................................:k000OOkxd:.  ...      .......................................
...................................'ccokkolxkkOxc. ........  .......................................
..................................;dllO0d.'dkxxxd; .........  ......................................
..................................:l:lxo'.,llllc:...........  ......................................
..............................''.....,do'.;ddc',l,.........   ......................................
.............................:OOkxkxxl:oxxk0kl;ok:....    ..........................................
.............................l0000000olOK0000O00O;           .......................................
.............................l000000Olo00000000KO;           .......................................
.............................l000000d:xK0KK0000KO;                ..................................
.............................'lolc:,..',;:clloodo'                   ...............................
............................                                         ...............................
..............................                                     .................................
.....................................                        .......................................
....................................................................................................
....................................................................................................
....................................................................................................
</pre>

How to use robots.txt parser

1

Paste your input

Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.
2

Click "Inspect"

host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.
3

Open the API tab

Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent

/api/v1/http/robots?q=https%3A%2F%2Fapi.security.github.com

curl -s '/api/v1/http/robots?q=https%3A%2F%2Fapi.security.github.com'

Embed this tool

<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?

Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.

Where does the data come from?

Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).

How fresh are the results?

Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.

Can I run this from the command line?

Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.

Can I monitor results over time?

Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.

host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors

✓Schedule any tool — every 1, 5, 15, 60 min
✓Diff against last run, alert on change
✓Webhook + email + Slack + PagerDuty + OpsGenie
✓Bulk CSV upload, 1,000 inputs per job
✓Export results as CSV / NDJSON / Excel
✓90-day history, comparison view