host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://dsinformatica.ar/robots.txt 404 9829 bytes 0 User-agent groups
Raw robots.txt 
<!DOCTYPE html><html lang="es" class="geist_a71539c9-module__T19VSG__variable geist_mono_8d43a2aa-module__8Li5zG__variable"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="/_next/static/chunks/08.a1gqz2-2~m.css" data-precedence="next"/><link rel="preload" as="script" fetchPriority="low" href="/_next/static/chunks/0t~1ydc1cx.~s.js"/><script src="/_next/static/chunks/03vcybz4ldxd4.js" async=""></script><script src="/_next/static/chunks/0m_p1bxtorv5i.js" async=""></script><script src="/_next/static/chunks/turbopack-01bxo9upvuwvx.js" async=""></script><script src="/_next/static/chunks/0g7ewx3bkljg5.js" async=""></script><script src="/_next/static/chunks/0d3tkj33oycf2.js" async=""></script><link rel="preload" href="https://www.googletagmanager.com/gtag/js?id=G-BJ6XKRCDEE" as="script"/><meta name="robots" content="noindex"/><meta name="next-size-adjust" content=""/><title>404: This page could not be found.</title><title>DS Informática | Soporte IT Empresarial</title><meta name="description" content="Soporte IT, servidores, redes, backups, monitoreo y seguridad informática para empresas."/><meta name="keywords" content="soporte IT,informática,redes,servidores,backups,seguridad informática,fortinet,pfsense,córdoba"/><link rel="icon" href="/favicon.ico?favicon.12vd6i355nkr8.ico" sizes="48x48" type="image/x-icon"/><script src="/_next/static/chunks/03~yq9q893hmn.js" noModule=""></script></head><body><div hidden=""><!--$--><!--/$--></div><div style="font-family:system-ui,&quot;Segoe UI&quot;,Roboto,Helvetica,Arial,sans-serif,&quot;Apple Color Emoji&quot;,&quot;Segoe UI Emoji&quot;;height:100vh;text-align:center;display:flex;flex-direction:column;align-items:center;justify-content:center"><div><style>body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}</style><h1 class="next-error-h1" style="display:inline-block;margin:0 20px 0 0;padding:0 23px 0 0;font-size:24px;font-weight:500;vertical-align:top;line-height:49px">404</h1><div style="display:inline-block"><h2 style="font-size:14px;font-weight:400;line-height:49px;margin:0">This page could not be found.</h2></div></div></div><!--$--><!--/$--><script src="/_next/static/chunks/0t~1ydc1cx.~s.js" id="_R_" async=""></script><script>(self.__next_f=self.__next_f||[]).push([0])</script><script>self.__next_f.push([1,"1:\"$Sreact.fragment\"\n2:I[79520,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"\"]\n3:I[39756,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"default\"]\n4:I[37457,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"default\"]\n5:I[97367,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"OutletBoundary\"]\n6:\"$Sreact.suspense\"\n9:I[97367,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"ViewportBoundary\"]\nb:I[97367,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"MetadataBoundary\"]\nd:I[68027,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"default\",1]\n:HL[\"/_next/static/chunks/08.a1gqz2-2~m.css\",\"style\"]\n"])</script><script>self.__next_f.push([1,"0:{\"P\":null,\"c\":[\"\",\"_not-found\"],\"q\":\"\",\"i\":false,\"f\":[[[\"\",{\"children\":[\"/_not-found\",{\"children\":[\"__PAGE__\",{}]}]},\"$undefined\",\"$undefined\",16],[[\"$\",\"$1\",\"c\",{\"children\":[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/chunks/08.a1gqz2-2~m.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\",\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-0\",{\"src\":\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"async\":true,\"nonce\":\"$undefined\"}],[\"$\",\"script\",\"script-1\",{\"src\":\"/_next/static/chunks/0d3tkj33oycf2.js\",\"async\":true,\"nonce\":\"$undefined\"}]],[\"$\",\"html\",null,{\"lang\":\"es\",\"className\":\"geist_a71539c9-module__T19VSG__variable geist_mono_8d43a2aa-module__8Li5zG__variable\",\"children\":[[\"$\",\"head\",null,{\"children\":[[\"$\",\"$L2\",null,{\"async\":true,\"src\":\"https://www.googletagmanager.com/gtag/js?id=G-BJ6XKRCDEE\"}],[\"$\",\"$L2\",null,{\"id\":\"google-analytics\",\"children\":\"\\n          window.dataLayer = window.dataLayer || [];\\n          function gtag(){dataLayer.push(arguments);}\\n          gtag('js', new Date());\\n\\n          gtag('config', 'G-BJ6XKRCDEE');\\n        \"}]]}],[\"$\",\"body\",null,{\"children\":[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":[[[\"$\",\"title\",null,{\"children\":\"404: This page could not be found.\"}],[\"$\",\"div\",null,{\"style\":{\"fontFamily\":\"system-ui,\\\"Segoe UI\\\",Roboto,Helvetica,Arial,sans-serif,\\\"Apple Color Emoji\\\",\\\"Segoe UI Emoji\\\"\",\"height\":\"100vh\",\"textAlign\":\"center\",\"display\":\"flex\",\"flexDirection\":\"column\",\"alignItems\":\"center\",\"justifyContent\":\"center\"},\"children\":[\"$\",\"div\",null,{\"children\":[[\"$\",\"style\",null,{\"dangerouslySetInnerHTML\":{\"__html\":\"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}\"}}],[\"$\",\"h1\",null,{\"className\":\"next-error-h1\",\"style\":{\"display\":\"inline-block\",\"margin\":\"0 20px 0 0\",\"padding\":\"0 23px 0 0\",\"fontSize\":24,\"fontWeight\":500,\"verticalAlign\":\"top\",\"lineHeight\":\"49px\"},\"children\":404}],[\"$\",\"div\",null,{\"style\":{\"display\":\"inline-block\"},\"children\":[\"$\",\"h2\",null,{\"style\":{\"fontSize\":14,\"fontWeight\":400,\"lineHeight\":\"49px\",\"margin\":0},\"children\":\"This page could not be found.\"}]}]]}]}]],[]],\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]}]]}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[null,[\"$\",\"$L3\",null,{\"parallelRouterKey\":\"children\",\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L4\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$undefined\",\"forbidden\":\"$undefined\",\"unauthorized\":\"$undefined\"}]]}],{\"children\":[[\"$\",\"$1\",\"c\",{\"children\":[[[\"$\",\"title\",null,{\"children\":\"404: This page could not be found.\"}],[\"$\",\"div\",null,{\"style\":\"$0:f:0:1:0:props:children:1:props:children:1:props:children:props:notFound:0:1:props:style\",\"children\":[\"$\",\"div\",null,{\"children\":[[\"$\",\"style\",null,{\"dangerouslySetInnerHTML\":{\"__html\":\"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}\"}}],[\"$\",\"h1\",null,{\"className\":\"next-error-h1\",\"style\":\"$0:f:0:1:0:props:children:1:props:children:1:props:children:props:notFound:0:1:props:children:props:children:1:props:style\",\"children\":404}],[\"$\",\"div\",null,{\"style\":\"$0:f:0:1:0:props:children:1:props:children:1:props:children:props:notFound:0:1:props:children:props:children:2:props:style\",\"children\":[\"$\",\"h2\",null,{\"style\":\"$0:f:0:1:0:props:children:1:props:children:1:props:children:props:notFound:0:1:props:children:props:children:2:props:children:props:style\",\"children\":\"This page could not be found.\"}]}]]}]}]],null,[\"$\",\"$L5\",null,{\"children\":[\"$\",\"$6\",null,{\"name\":\"Next.MetadataOutlet\",\"children\":\"$@7\"}]}]]}],{},null,false,null]},null,false,\"$@8\"]},null,false,null],[\"$\",\"$1\",\"h\",{\"children\":[[\"$\",\"meta\",null,{\"name\":\"robots\",\"content\":\"noindex\"}],[\"$\",\"$L9\",null,{\"children\":\"$La\"}],[\"$\",\"div\",null,{\"hidden\":true,\"children\":[\"$\",\"$Lb\",null,{\"children\":[\"$\",\"$6\",null,{\"name\":\"Next.Metadata\",\"children\":\"$Lc\"}]}]}],[\"$\",\"meta\",null,{\"name\":\"next-size-adjust\",\"content\":\"\"}]]}],false]],\"m\":\"$undefined\",\"G\":[\"$d\",[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/chunks/08.a1gqz2-2~m.css\",\"precedence\":\"next\",\"crossOrigin\":\"$undefined\",\"nonce\":\"$undefined\"}]]],\"S\":true,\"h\":null,\"s\":\"$undefined\",\"l\":\"$undefined\",\"p\":\"$undefined\",\"d\":\"$undefined\",\"b\":\"d9W2cn4vJgC4cACTbt7uA\"}\n"])</script><script>self.__next_f.push([1,"e:[]\n8:\"$We\"\n"])</script><script>self.__next_f.push([1,"a:[[\"$\",\"meta\",\"0\",{\"charSet\":\"utf-8\"}],[\"$\",\"meta\",\"1\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}]]\n"])</script><script>self.__next_f.push([1,"f:I[27201,[\"/_next/static/chunks/0g7ewx3bkljg5.js\",\"/_next/static/chunks/0d3tkj33oycf2.js\"],\"IconMark\"]\n7:null\nc:[[\"$\",\"title\",\"0\",{\"children\":\"DS Informática | Soporte IT Empresarial\"}],[\"$\",\"meta\",\"1\",{\"name\":\"description\",\"content\":\"Soporte IT, servidores, redes, backups, monitoreo y seguridad informática para empresas.\"}],[\"$\",\"meta\",\"2\",{\"name\":\"keywords\",\"content\":\"soporte IT,informática,redes,servidores,backups,seguridad informática,fortinet,pfsense,córdoba\"}],[\"$\",\"link\",\"3\",{\"rel\":\"icon\",\"href\":\"/favicon.ico?favicon.12vd6i355nkr8.ico\",\"sizes\":\"48x48\",\"type\":\"image/x-icon\"}],[\"$\",\"$Lf\",\"4\",{}]]\n"])</script></body></html>
Sponsored host.tools Pro
Bulk lookups · monitors · webhooks · 10,000 API calls/day

Upgrade to Pro for $19/mo. Cancel anytime. Works with the same API you already use.
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fdsinformatica.ar 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fdsinformatica.ar'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view