1. Crawler access
If GPTBot, ClaudeBot, PerplexityBot, or Google-Extended can't fetch your pages, none of the rest matters. Roughly one in three sites we scan blocks at least one major AI bot in robots.txt without realizing it. Add an llms.txt at the root to curate the pages you most want quoted — the full spec walkthrough is in our llms.txt guide.