Robots.txt Crawler Rules Parser

omnial/x402-robots-txt-parse

Given a website's URL, fetches and parses its robots.txt file and returns the crawl rules for every user-agent it names (allowed and disallowed paths, crawl delay), a total rule count, and whether the file blocks all crawlers outright -- useful for checking what a search engine or AI crawler is and is not permitted to access on a site.

activeper callv4

Price

Up to $0.0325 per call; you pay what the call used

You pay the price shown. If a call turns out to cost more than its quote, you pay the actual price, capped at twice the quote. No monthly minimum. Top-ups start at $5.00.

Data source

The requested web page

Data from the web page you request.

Use it

Your AI assistant does the work: it finds this tool, shows you the price first, runs it, and you are charged when the run finishes. There is nothing to set up for this tool in particular.

Connect once and your AI can find and run this tool, and any other it needs.

Install it Create an account

For developers: the calls your assistant makes

agent

// 1. search: find the tool
omnial_search({ query: "what you need" })

// 2. inspect: price and input schema
omnial_inspect({ tool: "omnial/x402-robots-txt-parse" })

// 3. learn: read the usage guide at runtime
omnial_learn({ tool: "omnial/x402-robots-txt-parse" })

// 4. run: price the exact call first, then run it
omnial_execute({ tool: "omnial/x402-robots-txt-parse", input: { /* see input_schema */ }, dry_run: true })
omnial_execute({ tool: "omnial/x402-robots-txt-parse", input: { /* ... */ } })

// 5. settle: the call is charged when it finishes, nothing to do

Not connected yet? Point your MCP client at https://mcp.eden-engine.com/mcp, then copy a config for your client.

Pricing

Every charge this tool can incur.

Charge	Price
Per call	$0.0325

Updated: Oct 27, 2026

Input and output schemas

Input schema

input_schema.json

{
  "type": "object",
  "required": [
    "url"
  ],
  "properties": {
    "url": {
      "type": "string",
      "description": "The website URL to fetch robots.txt for, e.g. 'https://example.com'."
    }
  },
  "additionalProperties": false
}

Output schema

output_schema.json

{
  "type": "object",
  "properties": {
    "rules": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "user_agent",
          "allow",
          "disallow"
        ],
        "properties": {
          "allow": {
            "type": "array",
            "items": {
              "type": "string"
            },
            "description": "Path patterns this user-agent is explicitly allowed. Often empty."
          },
          "disallow": {
            "type": "array",
            "items": {
              "type": "string"
            },
            "description": "Path patterns this user-agent is disallowed. Often empty."
          },
          "user_agent": {
            "type": "string"
          },
          "crawl_delay": {
            "type": "number",
            "description": "Seconds between requests this user-agent is asked to wait. Only present on rules that declare one -- observed on 'SemrushBot' (5), 'PerplexityBot' (1), and 'baidu' (1); absent (not null) on every other rule observed."
          }
        },
        "additionalProperties": true
      },
      "description": "One entry per named user-agent, in file order."
    },
    "exists": {
      "type": "boolean",
      "description": "Whether the site served a robots.txt file at all. Observed true in testing."
    },
    "raw_length": {
      "type": "number",
      "description": "Byte length of the raw robots.txt file that was fetched and parsed."
    },
    "total_rules": {
      "type": "number",
      "description": "Total number of per-user-agent rule blocks parsed (34 and 13 observed)."
    },
    "user_agents": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Every user-agent named in the file, in file order, including the wildcard '*' entry when present."
    },
    "total_allow_rules": {
      "type": "number",
      "description": "Total count of allow-path entries across all rules."
    },
    "blocks_all_crawlers": {
      "type": "boolean",
      "description": "True when the file's rules disallow every crawler outright. Observed false in testing; a genuinely blocking robots.txt has not been observed."
    },
    "total_disallow_rules": {
      "type": "number",
      "description": "Total count of disallow-path entries across all rules."
    }
  },
  "description": "The parsed robots.txt result. The shape when a site has no robots.txt file is not documented.",
  "additionalProperties": true
}