Robots, begone (#1812)

* robots.txt: "Please, don't come in." If a page is directly linked, the
  URL can still appear in search results, though.
* X-Robots-Tag: "If you're here, forget what you saw." Works even if the
  crawler ignores robots.txt or reaches a page via external link. Can
  remove already-indexed pages.
* Public boards may not be indexed. They're meant for "anyone with the
  link" private sharing, not worldwide publishing.

This commit is contained in:

Jeremy Daer

2025-12-02 13:35:58 -08:00

committed by

GitHub

parent 72f87f2d8f

commit b755b3fead

4 changed files with 45 additions and 1 deletions

public/robots.txt

+2 -1

View File

@@ -1 +1,2 @@
 # See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
 User-Agent: *
 Disallow: /