robots.txt parser
HTTP /api/v1/http/robotsFetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.
https://cony.app/robots.txt
200
4073 bytes
0 User-agent groups
Raw robots.txt
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/png" href="/cony.png?v=3" />
<link rel="apple-touch-icon" href="/apple-touch-icon.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
<meta name="theme-color" content="#020504" />
<meta name="color-scheme" content="dark" />
<title>Cony | Intent-Based Social Matching</title>
<meta
name="description"
content="Cony is an intent-based social matching app. Describe the match you're looking for in your own words and get matched by meaning, not profiles. No swiping."
/>
<link rel="canonical" href="https://cony.app/" />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
rel="preload"
as="style"
href="https://fonts.googleapis.com/css2?family=Manrope:wght@400;500;600;700;800&family=Sora:wght@500;600;700;800&display=swap"
/>
<link
rel="stylesheet"
href="https://fonts.googleapis.com/css2?family=Manrope:wght@400;500;600;700;800&family=Sora:wght@500;600;700;800&display=swap"
/>
<meta property="og:type" content="website" />
<meta property="og:site_name" content="Cony" />
<meta property="og:title" content="Cony | Intent-Based Social Matching" />
<meta
property="og:description"
content="No swiping. Describe the match you're looking for — Cony finds people who fit your intent."
/>
<meta property="og:url" content="https://cony.app/" />
<meta property="og:image" content="https://cony.app/og-image.jpg" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="630" />
<meta property="og:locale" content="en_US" />
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content="Cony | Intent-Based Social Matching" />
<meta
name="twitter:description"
content="Describe the match you're looking for. Cony finds the right people by meaning, not profiles."
/>
<meta name="twitter:image" content="https://cony.app/og-image.jpg" />
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Organization",
"name": "Cony",
"url": "https://cony.app/",
"logo": "https://cony.app/cony.png",
"description": "Intent-based social matching app. Describe the match you're looking for and get matched by meaning."
},
{
"@type": "WebSite",
"name": "Cony",
"url": "https://cony.app/"
},
{
"@type": "SoftwareApplication",
"name": "Cony",
"applicationCategory": "SocialNetworkingApplication",
"operatingSystem": "iOS, Android",
"description": "AI-native social matching app that understands your intent and helps you find the right people.",
"offers": { "@type": "Offer", "price": "0", "priceCurrency": "USD" }
}
]
}
</script>
<style>
body {
margin: 0;
background: #020504;
}
.brand-mark {
display: inline-flex;
align-items: center;
justify-content: center;
width: 44px;
height: 44px;
border-radius: 50%;
overflow: hidden;
background: #000;
}
.brand-mark img {
width: 100%;
height: 100%;
display: block;
object-fit: cover;
}
.nav-status,
.nav-status * {
color: #bfffe0;
-webkit-text-fill-color: #bfffe0;
}
</style>
<script type="module" crossorigin src="/assets/index-5JAMt-aR.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-DdE4ZOy7.css">
</head>
<body>
<a class="skip-link" href="#main">Skip to content</a>
<div id="root"></div>
</body>
</html>
How to use robots.txt parser
-
1
Paste your input
Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.
-
2
Click "Inspect"
host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.
-
3
Open the API tab
Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.
Why this matters
Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.
API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fcony.app
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fcony.app'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
width="100%" height="600" frameborder="0"></iframe>
Drop into any HTML page. The embed=1 flag hides nav and footer.
FAQ · robots.txt parser
Common questions
Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass
?nocache=1 for a forced refresh.Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI:
host.tools http robots YOUR_INPUT.Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro
Run robots.txt parser on a schedule. Get pinged when it changes.
Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.
- ✓Schedule any tool — every 1, 5, 15, 60 min
- ✓Diff against last run, alert on change
- ✓Webhook + email + Slack + PagerDuty + OpsGenie
- ✓Bulk CSV upload, 1,000 inputs per job
- ✓Export results as CSV / NDJSON / Excel
- ✓90-day history, comparison view