Jeremy Daer
b755b3fead
* robots.txt: "Please, don't come in." If a page is directly linked, the URL can still appear in search results, though. * X-Robots-Tag: "If you're here, forget what you saw." Works even if the crawler ignores robots.txt or reaches a page via external link. Can remove already-indexed pages. * Public boards may not be indexed. They're meant for "anyone with the link" private sharing, not worldwide publishing.
15 lines
340 B
Ruby
15 lines
340 B
Ruby
# Tell crawlers like Googlebot to drop pages entirely from search results, even
|
|
# if other sites link to it
|
|
module BlockSearchEngineIndexing
|
|
extend ActiveSupport::Concern
|
|
|
|
included do
|
|
after_action :block_search_engine_indexing
|
|
end
|
|
|
|
private
|
|
def block_search_engine_indexing
|
|
headers["X-Robots-Tag"] = "none"
|
|
end
|
|
end
|