# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file

User-agent: *
Allow: /

# Only what a crawler has no business fetching at all: the admin console and a
# signed-in user's own settings. Both sit behind authentication and neither is
# linked from a public page, so a crawl request is pure waste.
#
# Everything else that should stay out of the index says so in its own response
# instead — `noindex` in a <meta> tag on the auth and signup pages, the same in
# an X-Robots-Tag header on /learn/search, /learn/search_index and every page
# of the public demo organization. Disallowing those here did the opposite of
# what it looked like: a page Google cannot fetch is a page whose noindex
# Google never reads, and the search index the hub's search box fetches on
# every page turned into a "Blocked by robots.txt" report in Search Console.
#
# That applies to the demo more than anything else here. Its dashboard is
# linked from /demo, so crawlers will find it either way, and its shipment URLs
# are rebuilt nightly and 404 by morning. Blocked, those URLs could still be
# indexed from the link alone, with no content and no way to read the noindex.
# Crawlable, every one of them answers "noindex, nofollow" and stays out. Keep
# /demo itself indexable: it is the page meant to rank.
Disallow: /settings
Disallow: /admin/

# Meta's AI-training crawler, not a search engine: it sends no visitors. On
# 2026-10-02 it walked every include/exclude label combination of the demo's
# shipment filters (/es/northwind-produce/shipments?include_labels[]=…), about
# 640 requests in 15 minutes, and used up a rate limit our users shared with it.
User-agent: meta-externalagent
Disallow: /

# Sitemap
Sitemap: https://trackberry.com/sitemap.xml
Sitemap: https://trackberry.com/data/sitemap.xml