赋能 AI 代理：Robots.Txt (ETL-D API)

Large Language Models (LLMs) excel at generating human-like text and performing tasks but are prone to hallucinations, producing plausible yet incorrect or nonsensical responses. To address this issue, it is essential to integrate deterministic tools that guide their decision-making processes with reliable external data. The /robots.txt endpoint serves as such a tool, providing precise and authoritative information on how web crawlers interact with websites, thus helping LLMs avoid making incorrect assumptions about accessible data. This integration enables AI agents to process information more effectively and reduces the likelihood of erroneous outputs.