machinereadable
    established

    robots.txt

    The one file every crawler — search or AI — actually respects.

    robots.txt

    RFC 9309. The only file here with universal, decades-proven enforcement.

    ← Previous
    Next →

    ← Back to AI Site Files summary · Read the full guide for this file