# ============================================================
# robots.txt — MOSO.VN (Real Estate Platform, Next.js)
# Optimized for: Search Engines · AI Search (AIO/GEO/AEO) · LLMs
# Last Updated: 2026-07-07
#
# STRUCTURE NOTES (read before editing):
#
# 1. Single `User-agent: *` group. Per the Robots Exclusion Protocol
#    (RFC 9309) a crawler obeys ONLY its most-specific matching group,
#    so a named group like `User-agent: Googlebot` + `Allow: /` would
#    EXEMPT that bot from every rule below. All rules live in the one
#    default group so they reach every bot equally:
#      - Search: Googlebot, Bingbot, CocCocBot, DuckDuckBot, PetalBot,
#        YandexBot, Baiduspider, Yeti (Naver), Applebot, ...
#      - AI search & citation: GPTBot, OAI-SearchBot, ChatGPT-User,
#        ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot,
#        Google-Extended, Meta-ExternalAgent, Amazonbot, CCBot, ...
#      - SEO tools: AhrefsBot, SemrushBot, Screaming Frog, ...
#    All of the above are intentionally ALLOWED to crawl.
#
# 2. The group below is kept CONTIGUOUS — no blank or comment lines
#    inside it. Naive first-match parsers (Python robotparser and the
#    many crawlers built on it) treat a blank/comment line as the end
#    of the group and drop every rule after it.
#
# 3. No bare `Allow: /` — allow-all is the protocol default, and a
#    leading allow-all makes first-match parsers ignore all Disallows.
#    Allow exceptions are listed BEFORE their matching Disallow so
#    both first-match and longest-match parsers resolve them the same
#    way. Current exceptions:
#      - /api/meta-real-estate-feed stays open (Facebook catalog feed),
#        /api/agent/ (agent endpoints declared in the WebMCP manifest —
#        read endpoints plus the single POST-only write,
#        /api/agent/contact, which no crawler can trigger with a GET),
#        /api/markdown (page → markdown converter) and
#        /api/openapi.json (OpenAPI 3.1 spec for those endpoints) stay
#        open, while all other /api/ routes are blocked.
#      - /register-agent stays open while /register is blocked.
#
# 4. Content-Signal (contentsignals.org) declares how content may be
#    used: search=yes (index & cite in search / AI answers),
#    ai-input=yes (AI assistants may read pages to answer questions),
#    ai-train=no (do NOT use content to train foundation models).
#
# 5. Tracking parameters (utm_*, fbclid, gclid, ...) are intentionally
#    NOT disallowed. Every page emits a clean rel=canonical without
#    query params, which consolidates ranking signals from campaign
#    links and lets search/AI bots read pages shared with UTM tags.
#    Blocking them here would break that consolidation and cause
#    "indexed, though blocked" URLs. Client-side UTM analytics
#    (sessionStorage + /api/tracking) is unaffected by robots rules.
#
# 6. Pages carrying a noindex meta tag (e.g. /tim-kiem-nha-dat,
#    /xac-thuc-so-dien-thoai) are NOT blocked here — crawlers must be
#    able to fetch them to see the noindex directive.
#
# Disallowed below: Next.js API routes, auth/verification flows,
# private account areas, and post-login listing-management steps.
# ============================================================

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /llms.txt
Allow: /llms-full.txt
Allow: /llms-faq.txt
Allow: /api/meta-real-estate-feed
Allow: /api/agent/
Allow: /api/markdown
Allow: /api/openapi.json
Disallow: /api/
Allow: /register-agent
Disallow: /login
Disallow: /register
Disallow: /reset-password
Disallow: /oauth
Disallow: /email-verify
Disallow: /phone-verify
Disallow: /user-verify-phone
Disallow: /mine
Disallow: /quan-ly
Disallow: /profile
Disallow: /setting-account
Disallow: /tin-nhan
Disallow: /tim-kiem-da-luu
Disallow: /danh-sach-lich-hen
Disallow: /chi-tiet-khach-hang
Disallow: /unsubscribe-email
Disallow: /hoan-tat-tin-dang
Disallow: /gia-han-tin-dang
Disallow: /chap-nhan-tin-dang
# Cloudflare email-obfuscation artifact (/cdn-cgi/l/email-protection) — 404s on
# origin and is never a real page; keep crawlers out of the whole CF namespace.
Disallow: /cdn-cgi/

# ============================================================
# AI / LLM CONTENT FILES (llmstxt.org convention)
# Structured content for AI assistants, answer engines & LLM
# crawlers (ChatGPT, Claude, Gemini, Perplexity, ...). Explicitly
# allowed in the group above. Fetch these before crawling HTML:
#   https://moso.vn/llms.txt       — platform overview, citation
#                                    rules & keyword mapping
#   https://moso.vn/llms-full.txt  — full deep-dive documentation
#                                    (products, data, verification)
#   https://moso.vn/llms-faq.txt   — structured Q&A for AEO /
#                                    featured snippets / AI overview
# ============================================================

# ============================================================
# SITEMAPS & FEEDS
# ============================================================
# sitemap.xml is an index over sitemap-0.xml (static routes, generated from the
# App Router tree at build time) and sitemap-N.xml (CMS pages). The former
# hand-written sitemap-pages.xml is gone — its URLs are in sitemap-0.xml now.
# sitemap-news.xml: Google News feed (articles published in the last 48h).
# sitemap-ai.xml: curated factual/high-value pages for AI search & citation
# bots — mirrors the primary resources in llms.txt.
Sitemap: https://moso.vn/sitemap.xml
Sitemap: https://moso.vn/sitemap-news.xml
Sitemap: https://moso.vn/sitemap-ai.xml
Sitemap: https://moso.vn/rss.xml
Sitemap: https://moso.vn/tin-tuc-bat-dong-san/rss.xml