# robots.txt for www.getarbol.com (Arbol AI)
# The whole site is open: search engines, assistants and AI companies may read, cite,
# index and train on this content. No crawler is blocked.
#
# One group holds every crawler. The search and AI crawlers are named so each one finds
# itself listed; their rules are the same as everyone else's.
#
# Content-Signal (https://contentsignals.org/) says yes to every use:
#   search    building a search index and providing search results (links and short excerpts)
#   ai-input  inputting content into AI models, e.g. retrieval or grounding for AI answers
#   ai-train  training or fine-tuning AI models
# Content-Signal and Schemamap are extensions; crawlers that do not support them ignore them.

User-agent: *
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Googlebot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: Google-CloudVertexBot
User-agent: Bingbot
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: FacebookBot
User-agent: Amazonbot
User-agent: MistralAI-User
User-agent: cohere-ai
User-agent: DuckAssistBot
User-agent: DuckDuckBot
User-agent: YouBot
User-agent: Bytespider
User-agent: CCBot
User-agent: AI2Bot
User-agent: Diffbot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

Sitemap: https://www.getarbol.com/sitemap-index.xml
Schemamap: https://www.getarbol.com/schemamap.xml

# For agents: https://www.getarbol.com/llms.txt | https://www.getarbol.com/llms-full.txt | https://www.getarbol.com/agents.md | https://www.getarbol.com/auth.md | https://www.getarbol.com/ai.txt
# Every page has a markdown version: /path/ -> /path.md (the home page: /index.md)
# Blog feeds: https://www.getarbol.com/rss.xml | https://www.getarbol.com/en/rss.xml