How to Add llms.txt and AI Search Bot Rules (OAI-SearchBot, PerplexityBot, ClaudeBot) to Any Website

2026-11-21来源 dev.toEnglish

Traditional sitemap.xml files list every URL on your domain for search engine crawlers, whereas llms.txt is a lightweight Markdown map placed at your site root (yourdomain.com/llms.txt) that helps AI assistants and coding agents read your core documentation without HTML navigation clutter.

1. Separating AI Search Citations from Model Training in robots.txt

If you want your technical guides cited in ChatGPT Search, Perplexity, or Claude answers while controlling model pre-training access, split your user-agent rules:

 text User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: GPTBot Disallow: / 

原文:https://www.a8q9.cn/en/article/how-to-add-llms-txt-and-ai-search-bot-rules-oai-searchbot-perplexitybot-claudebo-cca729.html

相关文章