robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools

Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector

https://github.com/robots.txt 200 2274 bytes 5 User-agent groups

User-agent: *

Allow

/*?tab=achievements&achievement=*

Disallow

/*/*/pulse
/*/*/projects
/*/*/forks
/*/*/issues/new
/*/*/milestones/new
/*/*/issues/search
/*/*/commits/
/*/*/branches
/*/*/contributors
/*/*/tags
/*/*/stargazers
/*/*/watchers
/*/*/network
/*/*/graphs
/*/*/compare
/*/tree/
/gist/
/*/download
/*/revisions
/*/commits/*?author
/*/commits/*?path
/*/comments
/*/archive/
/*/blame/
/*/raw/
/*/cache/
/.git/
*/.git/
/*.git$
/search/advanced
/search$
/*q=
/*.atom$
/ekansa/Open-Context-Data
/ekansa/opencontext-*
*/tarball/
*/zipball/
/*source=*
/*ref_cta=*
/*plan=*
/*return_to=*
/*ref_loc=*
/*setup_organization=*
/*source_repo=*
/*ref_page=*
/*source=*
/*referrer=*
/*report=*
/*author=*
/*since=*
/*until=*
/*commits?author=*
/*report-abuse?report=*
/*tab=*
/account-login
/Explodingstuff/
/copilot/
/copilot/c/

User-agent: *

Allow

/*?tab=achievements&achievement=*

Disallow

/*/*/pulse
/*/*/projects
/*/*/forks
/*/*/issues/new
/*/*/milestones/new
/*/*/issues/search
/*/*/commits/
/*/*/branches
/*/*/contributors
/*/*/tags
/*/*/stargazers
/*/*/watchers
/*/*/network
/*/*/graphs
/*/*/compare
/*/tree/
/gist/
/*/download
/*/revisions
/*/commits/*?author
/*/commits/*?path
/*/comments
/*/archive/
/*/blame/
/*/raw/
/*/cache/
/.git/
*/.git/
/*.git$
/search/advanced
/search$
/*q=
/*.atom$
/ekansa/Open-Context-Data
/ekansa/opencontext-*
*/tarball/
*/zipball/
/*source=*
/*ref_cta=*
/*plan=*
/*return_to=*
/*ref_loc=*
/*setup_organization=*
/*source_repo=*
/*ref_page=*
/*source=*
/*referrer=*
/*report=*
/*author=*
/*since=*
/*until=*
/*commits?author=*
/*report-abuse?report=*
/*tab=*
/account-login
/Explodingstuff/
/copilot/
/copilot/c/

User-agent: *

Allow

/*?tab=achievements&achievement=*

Disallow

/*/*/pulse
/*/*/projects
/*/*/forks
/*/*/issues/new
/*/*/milestones/new
/*/*/issues/search
/*/*/commits/
/*/*/branches
/*/*/contributors
/*/*/tags
/*/*/stargazers
/*/*/watchers
/*/*/network
/*/*/graphs
/*/*/compare
/*/tree/
/gist/
/*/download
/*/revisions
/*/commits/*?author
/*/commits/*?path
/*/comments
/*/archive/
/*/blame/
/*/raw/
/*/cache/
/.git/
*/.git/
/*.git$
/search/advanced
/search$
/*q=
/*.atom$
/ekansa/Open-Context-Data
/ekansa/opencontext-*
*/tarball/
*/zipball/
/*source=*
/*ref_cta=*
/*plan=*
/*return_to=*
/*ref_loc=*
/*setup_organization=*
/*source_repo=*
/*ref_page=*
/*source=*
/*referrer=*
/*report=*
/*author=*
/*since=*
/*until=*
/*commits?author=*
/*report-abuse?report=*
/*tab=*
/account-login
/Explodingstuff/
/copilot/
/copilot/c/

User-agent: *

Allow

/*?tab=achievements&achievement=*

Disallow

/*/*/pulse
/*/*/projects
/*/*/forks
/*/*/issues/new
/*/*/milestones/new
/*/*/issues/search
/*/*/commits/
/*/*/branches
/*/*/contributors
/*/*/tags
/*/*/stargazers
/*/*/watchers
/*/*/network
/*/*/graphs
/*/*/compare
/*/tree/
/gist/
/*/download
/*/revisions
/*/commits/*?author
/*/commits/*?path
/*/comments
/*/archive/
/*/blame/
/*/raw/
/*/cache/
/.git/
*/.git/
/*.git$
/search/advanced
/search$
/*q=
/*.atom$
/ekansa/Open-Context-Data
/ekansa/opencontext-*
*/tarball/
*/zipball/
/*source=*
/*ref_cta=*
/*plan=*
/*return_to=*
/*ref_loc=*
/*setup_organization=*
/*source_repo=*
/*ref_page=*
/*source=*
/*referrer=*
/*report=*
/*author=*
/*since=*
/*until=*
/*commits?author=*
/*report-abuse?report=*
/*tab=*
/account-login
/Explodingstuff/
/copilot/
/copilot/c/

User-agent: *

Allow

/*?tab=achievements&achievement=*

Disallow

/*/*/pulse
/*/*/projects
/*/*/forks
/*/*/issues/new
/*/*/milestones/new
/*/*/issues/search
/*/*/commits/
/*/*/branches
/*/*/contributors
/*/*/tags
/*/*/stargazers
/*/*/watchers
/*/*/network
/*/*/graphs
/*/*/compare
/*/tree/
/gist/
/*/download
/*/revisions
/*/commits/*?author
/*/commits/*?path
/*/comments
/*/archive/
/*/blame/
/*/raw/
/*/cache/
/.git/
*/.git/
/*.git$
/search/advanced
/search$
/*q=
/*.atom$
/ekansa/Open-Context-Data
/ekansa/opencontext-*
*/tarball/
*/zipball/
/*source=*
/*ref_cta=*
/*plan=*
/*return_to=*
/*ref_loc=*
/*setup_organization=*
/*source_repo=*
/*ref_page=*
/*source=*
/*referrer=*
/*report=*
/*author=*
/*since=*
/*until=*
/*commits?author=*
/*report-abuse?report=*
/*tab=*
/account-login
/Explodingstuff/
/copilot/
/copilot/c/

Raw robots.txt

# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots
# We also provide an extensive API: https://docs.github.com
User-agent: bingbot
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: adidxbot
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: BingPreview
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: baidu
crawl-delay: 1


User-agent: *

Disallow: /*/*/pulse
Disallow: /*/*/projects
Disallow: /*/*/forks
Disallow: /*/*/issues/new
Disallow: /*/*/milestones/new
Disallow: /*/*/issues/search
Disallow: /*/*/commits/
Disallow: /*/*/branches
Disallow: /*/*/contributors
Disallow: /*/*/tags
Disallow: /*/*/stargazers
Disallow: /*/*/watchers
Disallow: /*/*/network
Disallow: /*/*/graphs
Disallow: /*/*/compare

Disallow: /*/tree/
Disallow: /gist/
Disallow: /*/download
Disallow: /*/revisions
Disallow: /*/commits/*?author
Disallow: /*/commits/*?path
Disallow: /*/comments
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/raw/
Disallow: /*/cache/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /search/advanced
Disallow: /search$
Disallow: /*q=
Disallow: /*.atom$

Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: */tarball/
Disallow: */zipball/

Disallow: /*source=*
Disallow: /*ref_cta=*
Disallow: /*plan=*
Disallow: /*return_to=*
Disallow: /*ref_loc=*
Disallow: /*setup_organization=*
Disallow: /*source_repo=*
Disallow: /*ref_page=*
Disallow: /*source=*
Disallow: /*referrer=*
Disallow: /*report=*
Disallow: /*author=*
Disallow: /*since=*
Disallow: /*until=*
Disallow: /*commits?author=*
Disallow: /*report-abuse?report=*
Disallow: /*tab=*
Allow: /*?tab=achievements&achievement=*

Disallow: /account-login
Disallow: /Explodingstuff/

Disallow: /copilot/
Disallow: /copilot/c/

How to use robots.txt parser

1

Paste your input

Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.
2

Click "Inspect"

host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.
3

Open the API tab

Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent

/api/v1/http/robots?q=https%3A%2F%2Fgithub.com

curl -s '/api/v1/http/robots?q=https%3A%2F%2Fgithub.com'

Embed this tool

<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?

Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.

Where does the data come from?

Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).

How fresh are the results?

Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.

Can I run this from the command line?

Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.

Can I monitor results over time?

Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.

host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors

✓Schedule any tool — every 1, 5, 15, 60 min
✓Diff against last run, alert on change
✓Webhook + email + Slack + PagerDuty + OpsGenie
✓Bulk CSV upload, 1,000 inputs per job
✓Export results as CSV / NDJSON / Excel
✓90-day history, comparison view