* robots.txt: "Please, don't come in." If a page is directly linked, the
URL can still appear in search results, though.
* X-Robots-Tag: "If you're here, forget what you saw." Works even if the
crawler ignores robots.txt or reaches a page via external link. Can
remove already-indexed pages.
* Public boards may not be indexed. They're meant for "anyone with the
link" private sharing, not worldwide publishing.
This is a great, solid alternative to CSRF tokens for CSRF protection
when we aren't worried about older browsers or other kind of actors
doing modifying requests in our app, and could be a good test for future
upstreaming to Rails (although there we'd need to continue using CSRF
tokens or at least letting people opt out manually).
Let's start checking the header and reporting on it when CSRF fails or
when it doesn't match the other checks Rails does, and then promote this
to be the only way to defend from CSRF.
Instead of writer pinning, we'll track the last transaction ID of each
write in the session. Then on each read we'll wait for the replica to
report that this transaction is available.
If it doesn't become available within a reasonable timeout, we'll
proceed anyway, and accept the possibility of a stale read.
The hope here is that most of the time, the replica is caught up in the
time between a write request and the following read request. If it's
not, we now have a little tolerance to wait for it, which hopefully
proves enough to stale reads are not encountered in normal use.
We also disable the writer affinity opt-out mechanism that we had
before, since we will no longer be using writer affinity at the load
balancer.
In order to control routing dynamically, we need to expose some Beamer
information to the proxy.
The `beamer_writer` value is used by the proxy to keep track of which
node should receive write requests. When it changes due to a failover,
the proxy will update after seeing a new value in this header. We
provide this to the proxy in the `X-Writer` response header.
The `beamer_last_txn` value will be used to control writer pinning.
Setting it in the cookie here is the first stage of this. The second
stage will be to catch situations where a reader gets a request where
this header value is set to a large value than the reader has seen; when
that happens if means there's a risk of reading stale data, so rather
than serve the request we should request the proxy to reproxy it back to
the writer.
We also set `X-Kamal-Target` in the response to match the
`X-Kamal-Target` that was set in the request; this lets us see which
proxy target served each request, which is useful for diagnostics.
Yes, Platform Agent is overkill for just this but there isn't a one-size JS solution for updating both input placeholders and other HTML text and we're likely to need it later for other things like displaying platform-specific PWA prompts.
Jeremy Daer