Robots, begone (#1812)

* robots.txt: "Please, don't come in." If a page is directly linked, the
  URL can still appear in search results, though.
* X-Robots-Tag: "If you're here, forget what you saw." Works even if the
  crawler ignores robots.txt or reaches a page via external link. Can
  remove already-indexed pages.
* Public boards may not be indexed. They're meant for "anyone with the
  link" private sharing, not worldwide publishing.

app/controllers/application_controller.rb

@@ -1,6 +1,7 @@
 class ApplicationController < ActionController::Base
   include Authentication
   include Authorization
+  include BlockSearchEngineIndexing
   include CurrentRequest, CurrentTimezone, SetPlatform
   include RequestForgeryProtection
   include TurboFlash, ViewTransitions

app/controllers/concerns/block_search_engine_indexing.rb

@@ -0,0 +1,14 @@
+# Tell crawlers like Googlebot to drop pages entirely from search results, even
+# if other sites link to it
+module BlockSearchEngineIndexing
+  extend ActiveSupport::Concern
+
+  included do
+    after_action :block_search_engine_indexing
+  end
+
+  private
+    def block_search_engine_indexing
+      headers["X-Robots-Tag"] = "none"
+    end
+end