* robots.txt: "Please, don't come in." If a page is directly linked, the
URL can still appear in search results, though.
* X-Robots-Tag: "If you're here, forget what you saw." Works even if the
crawler ignores robots.txt or reaches a page via external link. Can
remove already-indexed pages.
* Public boards may not be indexed. They're meant for "anyone with the
link" private sharing, not worldwide publishing.
This is a great, solid alternative to CSRF tokens for CSRF protection
when we aren't worried about older browsers or other kind of actors
doing modifying requests in our app, and could be a good test for future
upstreaming to Rails (although there we'd need to continue using CSRF
tokens or at least letting people opt out manually).
Let's start checking the header and reporting on it when CSRF fails or
when it doesn't match the other checks Rails does, and then promote this
to be the only way to defend from CSRF.
Jeremy Daer