{
"domain": "example.com",
"robotsTxt": {
"url": "https://example.com/robots.txt",
"found": false,
"statusCode": 404,
"groups": [],
"declaredSitemaps": []
},
"sitemap": {
"source": "fallback_sitemap_xml",
"results": [
{
"url": "https://example.com/sitemap.xml",
"found": false,
"statusCode": 404
}
]
},
"issues": [
{
"code": "robots_missing",
"severity": "info",
"message": "robots.txt가 없습니다 — 기본적으로 전체 크롤링이 허용됩니다."
},
{
"code": "no_sitemap_declared",
"severity": "info",
"message": "robots.txt에 Sitemap 필드가 선언되어 있지 않습니다 — /sitemap.xml 기본 경로만 확인했습니다."
}
]
}
curl --location --request GET 'https://zylalabs.com/api/14047/robots+and+sitemap+checker+api/34290/check+robots+txt+and+sitemap' --header 'Authorization: Bearer YOUR_API_KEY'
After signing up, every developer is assigned a personal API access key, a unique combination of letters and digits provided to access to our API endpoint. To authenticate with the Robots and Sitemap Checker API simply include your bearer token in the Authorization header.
| Header | Description |
|---|---|
Authorization
|
Required
Should be Bearer access_key. See "Your API Access Key" above when you are subscribed.
|
No long-term commitment. Upgrade, downgrade, or cancel anytime. Free Trial includes up to 50 requests.
Given a single domain, checks that site's robots.txt (crawl allow/disallow rules per user-agent, crawl-delay, declared sitemap URLs) and sitemap.xml (existence, URL count, and child-sitemap count if
The endpoint returns structured JSON data that includes the status of the robots.txt file, sitemap URLs, and any detected issues. It provides information on crawl rules, sitemap existence, URL counts, and flags common mistakes.
Key fields include `domain`, `robotsTxt` (with `url`, `found`, `statusCode`, `groups`, and `declaredSitemaps`), `sitemap` (with `source` and `results`), and `issues` (with `code`, `severity`, and `message`).
The response is organized into three main sections: `domain` for the target site, `robotsTxt` for crawl rules and sitemap declarations, and `sitemap` for sitemap status. Each section contains relevant details and any identified issues.
The endpoint provides information on the existence and status of the robots.txt file, declared sitemaps, URL counts in the sitemap, and any issues like missing files or crawl blocks.
Users can customize requests by specifying the target domain in the API call. The endpoint does not require additional parameters, making it straightforward to use for any domain.
Typical use cases include SEO audits, verifying site migrations, and conducting pre-deployment checks to ensure proper crawl settings and sitemap accessibility.
Data accuracy is maintained by adhering to the RFC 9309 standard for robots.txt parsing and by systematically checking the existence and status of the sitemap URLs.
If results are partial or empty, users should check the `found` field in the response. If `false`, it indicates that the robots.txt or sitemap was not found, prompting further investigation into the domain's configuration.
To obtain your API key, first sign in to your account and navigate to the API you want to use. From the API's Pricing section, choose a plan and complete the subscription process. Once subscribed, return to the API page and you will see your API Access Key displayed at the top of the documentation page. You can use this key to authenticate your requests.
You can’t switch APIs during the free trial. If you subscribe to a different API, your trial will end and the new subscription will start as a paid plan.
The free trial lasts for 7 days and allows you to make up to 50 API requests.
No, the free trial is available only once, so we recommend using it on the API that interests you the most. Most of our APIs offer a free trial, but some may not include this option.
Yes. If the API offers a free trial, you will see a "Free 7-Day Trial" option in its Pricing section. The trial lasts for 7 days and allows up to 50 API requests, enabling you to evaluate the API before subscribing to a paid plan.
Zyla API Hub is like a big store for APIs, where you can find thousands of them all in one place. We also offer dedicated support and real-time monitoring of all APIs. Once you sign up, you can pick and choose which APIs you want to use. Just remember, each API needs its own subscription. But if you subscribe to multiple ones, you'll use the same key for all of them, making things easier for you.
You can monitor your API usage through the response headers included with every request:
x-zyla-api-calls-monthly-used: Shows the total number of API requests you have used during the current billing period.
x-zyla-api-calls-monthly-remaining: Shows the number of API requests you have remaining for the current billing period.
Yes, you can cancel your subscription at any time. Simply go to the Pricing section of the API you're subscribed to and click the "Unsubscribe" button.
Please note that upgrades, downgrades, and cancellations take effect immediately. Once your subscription is canceled, access to the service will end immediately, regardless of any remaining API calls in your quota.
Please have a look at our Refund Policy: https://zylalabs.com/terms#refund