📄 中文摘要
配置 robots.txt 文件可以控制特定爬虫对网站的访问权限。示例配置中，针对多种爬虫（如 GPTBot、CCBot 和 PerplexityBot）设置了禁止访问的规则。尽管如此，AI 仍然能够通过其他技术手段读取网站内容。这些技术向量包括但不限于使用 API、分析网页结构、利用缓存数据等。了解这些技术向量对于网站管理员和开发者至关重要，以便更好地保护网站内容和隐私。通过合理配置和监控，能够有效管理 AI 对网站的访问行为。
robots.txt 是一个信号，而不是围栏：AI 读取网站的八个技术向量
📄 English Summary
robots.txt is a sign, not a fence: 8 technical vectors through which AI still reads your website
The configuration of the robots.txt file allows control over specific crawlers' access to a website. The provided example includes disallow rules for various crawlers such as GPTBot, CCBot, and PerplexityBot. However, AI can still read website content through other technical vectors. These vectors include, but are not limited to, using APIs, analyzing webpage structures, and leveraging cached data. Understanding these technical vectors is crucial for website administrators and developers to better protect website content and privacy. Proper configuration and monitoring can effectively manage AI's access behavior to the website.