# robots.txt for NextMotion.net # Last updated: 2025-10-21 # Purpose: Control search engine and AI bot crawling # ============================================================================== # DEFAULT RULES FOR ALL BOTS # ============================================================================== User-agent: * # Allow public pages Allow: / Allow: /blog/ Allow: /about/ Allow: /pricing/ Allow: /contact_form/ Allow: /capture/ Allow: /consult/ Allow: /agenda/ Allow: /3d/ Allow: /revolution/ Allow: /ai/ Allow: /ai-copilot/ Allow: /starter/ Allow: /practitioner/ Allow: /clinic-manager/ Allow: /marketing/ Allow: /multi-site/ # Allow multilingual pages (FR, ES) — de/pt/he/ar éteintes (301 → EN) Allow: /fr/ Allow: /es/ # Block private and protected areas Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ # Block API endpoints Disallow: /api/ # Public JavaScript, CSS and optimized images must remain crawlable for rendering. Allow: /_next/static/ Allow: /_next/image/ # Block admin areas Disallow: /admin/ Disallow: /dashboard/ # Block authentication pages (if any) Disallow: /login/ Disallow: /signup/ Disallow: /auth/ # Block temporary or test pages Disallow: /test/ Disallow: /temp/ Disallow: /staging/ # Crawl-delay for general bots (optional) # Crawl-delay: 10 # ============================================================================== # SITEMAPS # ============================================================================== # Main sitemap (EN root + FR legal pages) Sitemap: https://www.nextmotion.net/sitemap.xml # FR sitemap (redesign-aware: switches old->new slugs at the live flip) Sitemap: https://www.nextmotion.net/fr/sitemap.xml # ES sitemap (redesign-aware: switches old->new slugs at the live flip) Sitemap: https://www.nextmotion.net/es/sitemap.xml # Blog sitemap Sitemap: https://www.nextmotion.net/blog/sitemap.xml # ============================================================================== # GOOGLE BOT (Optimized crawling) # ============================================================================== User-agent: Googlebot Allow: / Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ Allow: /_next/static/ Disallow: /admin/ # No crawl-delay for Google (they optimize automatically) User-agent: Googlebot-Image Allow: / Disallow: /academy/ Disallow: /api/ User-agent: Googlebot-Video Allow: / Disallow: /academy/ Disallow: /api/ # Google's extended bot for AI training User-agent: Google-Extended Allow: / Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ # ============================================================================== # BING BOT # ============================================================================== User-agent: Bingbot Allow: / Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ Allow: /_next/static/ Disallow: /admin/ # ============================================================================== # AI BOTS (Language Models) # ============================================================================== # OpenAI GPT Bot User-agent: GPTBot Allow: / Allow: /blog/ Allow: /about/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ Disallow: /admin/ Crawl-delay: 10 # OpenAI search crawler (ChatGPT search) User-agent: OAI-SearchBot Allow: / Allow: /blog/ Allow: /about/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ # ChatGPT User (for browsing feature) User-agent: ChatGPT-User Allow: / Allow: /blog/ Allow: /about/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ Crawl-delay: 10 # Anthropic Claude (crawler d'entraînement, recherche et navigation utilisateur) User-agent: ClaudeBot Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ User-agent: Claude-User Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ User-agent: Claude-SearchBot Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ User-agent: Claude-Web Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ Crawl-delay: 10 User-agent: Anthropic-AI Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /api/ Crawl-delay: 10 # Perplexity AI User-agent: PerplexityBot Allow: / Allow: /blog/ Allow: /about/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ Crawl-delay: 10 User-agent: Perplexity-User Allow: / Allow: /blog/ Allow: /about/ Disallow: /academy/ Disallow: /fr/academy/ Disallow: /de/academy/ Disallow: /es/academy/ Disallow: /api/ # Common Crawl (used for AI training) User-agent: CCBot Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /api/ Crawl-delay: 15 # Omgili bot (news aggregator) User-agent: Omgilibot Allow: /blog/ Disallow: /academy/ Disallow: /api/ Crawl-delay: 10 # Apple Intelligence User-agent: Applebot-Extended Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /api/ Crawl-delay: 10 # Facebook Bot (Meta AI) User-agent: FacebookBot Allow: / Allow: /blog/ Disallow: /academy/ Disallow: /api/ User-agent: facebookexternalhit Allow: / # Bytespider (TikTok) - BLOCKED User-agent: Bytespider Disallow: / # ============================================================================== # OTHER SEARCH ENGINES # ============================================================================== # Yandex (Russian) User-agent: Yandex Allow: / Disallow: /academy/ Disallow: /api/ Disallow: /admin/ # Baidu (Chinese) User-agent: Baiduspider Allow: / Disallow: /academy/ Disallow: /api/ Crawl-delay: 10 # DuckDuckGo User-agent: DuckDuckBot Allow: / Disallow: /academy/ Disallow: /api/ # Yahoo/Slurp User-agent: Slurp Allow: / Disallow: /academy/ Disallow: /api/ # ============================================================================== # SOCIAL MEDIA BOTS # ============================================================================== # Twitter/X User-agent: Twitterbot Allow: / # LinkedIn User-agent: LinkedInBot Allow: / # Pinterest User-agent: Pinterestbot Allow: / Disallow: /academy/ Disallow: /api/ # WhatsApp (for link previews) User-agent: WhatsApp Allow: / # Telegram User-agent: TelegramBot Allow: / # ============================================================================== # SEO & ANALYTICS BOTS (Generally allowed) # ============================================================================== # Ahrefs User-agent: AhrefsBot Allow: / Crawl-delay: 10 # SEMrush User-agent: SemrushBot Allow: / Crawl-delay: 10 # Moz User-agent: rogerbot Allow: / Crawl-delay: 10 # Screaming Frog User-agent: Screaming Frog SEO Spider Allow: / # ============================================================================== # ARCHIVE BOTS # ============================================================================== # Internet Archive Wayback Machine User-agent: ia_archiver Allow: / Disallow: /academy/ Disallow: /api/ Disallow: /admin/ # ============================================================================== # BAD BOTS & SCRAPERS (BLOCKED) # ============================================================================== # Block aggressive scrapers User-agent: SemrushBot Crawl-delay: 10 User-agent: AhrefsBot Crawl-delay: 10 # Block known bad bots User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: SEOkicks Disallow: / User-agent: BLEXBot Disallow: / User-agent: SeznamBot Crawl-delay: 15 # Block email harvesters User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / # Block content scrapers User-agent: WebCopier Disallow: / User-agent: HTTrack Disallow: / User-agent: WebReaper Disallow: / User-agent: WebStripper Disallow: / # Block spam bots User-agent: SpamBot Disallow: / User-agent: spbot Disallow: / # ============================================================================== # SPECIAL NOTES # ============================================================================== # Academy Subdomain (academy.nextmotion.net): # This robots.txt is for the main site (www.nextmotion.net) # The academy subdomain should have its own robots.txt that blocks everything: # # User-agent: * # Disallow: / # # This is because academy.nextmotion.net is a private, login-required area. # LLMS.txt (guide du site pour les assistants IA, spec https://llmstxt.org/) : # https://www.nextmotion.net/llms.txt (index des pages + blog, Markdown) # https://www.nextmotion.net/llms-full.txt (FAQ, pages légales et articles en intégral) # Générés au build depuis le code (app/llms.txt, app/llms-full.txt), pas de fichier statique. # Security: # This robots.txt does NOT provide security. Use proper authentication, # authorization, and firewall rules to protect sensitive areas. # Monitoring: # Monitor bot traffic in server logs and Google Search Console to adjust # these rules as needed. # Contact: # For questions about crawling policies: support@nextmotion.net # ============================================================================== # END OF ROBOTS.TXT # ==============================================================================