robots.txt parser
HTTP /api/v1/http/robotsFetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.
https://liyuan.app/robots.txt
403
7449 bytes
0 User-agent groups
Raw robots.txt
<!doctype html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!-->
<html class="no-js" lang="en-US">
<!--<![endif]-->
<head>
<title>Edge IP Restricted | liyuan.app | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" /> <script>
(function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){var c=document.getElementById("error-feedback-survey"),d=document.getElementById("error-feedback-success"),b=new XMLHttpRequest;a={event:"feedback clicked",properties:{errorCode: 1034 },helpful:a,version: 1 };b.open("POST","https://sparrow.cloudflare.com/api/v1/event");b.setRequestHeader("Content-Type","application/json");b.setRequestHeader("Sparrow-Source-Key","c771f0e4b54944bebf4261d44bd79a1e");
b.send(JSON.stringify(a));c.classList.add("feedback-hidden");d.classList.remove("feedback-hidden")};document.addEventListener("DOMContentLoaded",function(){var a=document.getElementById("error-feedback"),c=document.getElementById("feedback-button-yes"),d=document.getElementById("feedback-button-no");"classList"in a&&(a.classList.remove("feedback-hidden"),c.addEventListener("click",function(){e(!0)}),d.addEventListener("click",function(){e(!1)}))})}})();
</script>
<script
defer
src="https://performance.radar.cloudflare.com/beacon.js"
></script>
</head>
<body>
<div id="cf-wrapper">
<div
class="cf-alert cf-alert-error cf-cookie-error hidden"
id="cookie-alert"
data-translate="enable_cookies"
>
Please enable cookies.
</div>
<div id="cf-error-details" class="p-0">
<header
class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased"
>
<h1
class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight"
>
<span data-translate="error">Error</span>
<span>1034</span>
</h1>
<span
class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed"
>Ray ID: 9f90cd5ace41b900 •</span
>
<span
class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed"
>2026-05-09 12:45:35 UTC</span
>
<h2
class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light"
>
Edge IP Restricted
</h2>
</header>
<section class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
<div id="what-happened-section" class="w-1/2 md:w-full">
<h2
class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased"
data-translate="what_happened"
>
What happened?
</h2>
<p>You've requested a page on a website that is part of the <a href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a> network. The host (liyuan.app) resolved to an IP address that the owner of the website does not have access to.</p>
</div>
<div
id="resolution-copy-section"
class="w-1/2 mt-6 text-15 leading-normal"
>
<h2
class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased"
data-translate="what_can_i_do"
>
What can I do?
</h2>
<p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Check your DNS records to ensure they are pointed to the IP address(es) you were assigned at registration.</p>
</div>
</section>
<div class="feedback-hidden py-8 text-center" id="error-feedback">
<div id="error-feedback-survey" class="footer-line-wrapper">
Was this page helpful?
<button
class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded"
id="feedback-button-yes"
type="button"
>
Yes
</button>
<button
class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded"
id="feedback-button-no"
type="button"
>
No
</button>
</div>
<div class="feedback-success feedback-hidden" id="error-feedback-success">
Thank you for your feedback!
</div>
</div> <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
<p class="text-13">
<span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">9f90cd5ace41b900</strong></span>
<span class="cf-footer-separator sm:hidden">•</span>
<span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
Your IP:
<button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
<span class="hidden" id="cf-footer-ip">45.87.80.250</span>
<span class="cf-footer-separator sm:hidden">•</span>
</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Performance & security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>
</p>
<script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->
</div>
<!-- /#cf-error-details -->
</div>
<!-- /#cf-wrapper -->
<script>
window._cf_translation = {};
</script>
</body>
</html>How to use robots.txt parser
-
1
Paste your input
Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.
-
2
Click "Inspect"
host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.
-
3
Open the API tab
Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.
Why this matters
Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.
API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fliyuan.app
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fliyuan.app'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
width="100%" height="600" frameborder="0"></iframe>
Drop into any HTML page. The embed=1 flag hides nav and footer.
FAQ · robots.txt parser
Common questions
Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass
?nocache=1 for a forced refresh.Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI:
host.tools http robots YOUR_INPUT.Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro
Run robots.txt parser on a schedule. Get pinged when it changes.
Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.
- ✓Schedule any tool — every 1, 5, 15, 60 min
- ✓Diff against last run, alert on change
- ✓Webhook + email + Slack + PagerDuty + OpsGenie
- ✓Bulk CSV upload, 1,000 inputs per job
- ✓Export results as CSV / NDJSON / Excel
- ✓90-day history, comparison view