* robots.txt: "Please, don't come in." If a page is directly linked, the
URL can still appear in search results, though.
* X-Robots-Tag: "If you're here, forget what you saw." Works even if the
crawler ignores robots.txt or reaches a page via external link. Can
remove already-indexed pages.
* Public boards may not be indexed. They're meant for "anyone with the
link" private sharing, not worldwide publishing.
Jeremy Daer
b755b3fead