An append-only storage ledger replaces deadlock-prone synchronous
counter updates and drift-prone async updates.
All content storage (card images, card/comment/board description embeds)
is tracked. Account exports (which expire) and avatars are not tracked.
Storage is accounted for by Account and Board. Both consume the same
event stream independently: no bubble-up storage bumps triggering
deadlocks due to lock sequencing. Each calculates its own total from
the underlying ledger with independent cursors and materialization.
* Storage::Entry: Append-only ledger recording attach/detach/transfer
events with delta bytes. Single event stream indexed for both
Account and Board cursor queries.
* Storage::Total: Polymorphic snapshot cache with cursor (last_entry_id)
tracking which entries have been materialized.
* Storage::Totaled concern: Provides bytes_used (fast snapshot) and
bytes_used_exact (snapshot + pending) query modes, plus
materialize_storage! to roll up pending entries.
* Storage::Tracked concern: For models owning attachments (Card,
Comment, Board). Provides board_for_storage_tracking for models
where board is determined differently (Board returns self).
Handles board transfers by recording transfer_out/transfer_in entries.
* Storage::AttachmentTracking: Hooks ActiveStorage::Attachment lifecycle
to record attach/detach entries. Handles ActionText::RichText embeds
by traversing to the actual model. Snapshots context in before_destroy
to handle cascading deletes where parent record may be gone by
after_destroy_commit.
* MaterializeJob: Rolls up pending entries into snapshot. Concurrency
limited per owner to prevent duplicate work.
* ReconcileJob: On-demand reconciliation against actual attachment
storage for support/debugging. Compares ledger total to real bytes
from card images, card embeds, comment embeds, and board embeds.
Usage:
* account.bytes_used / board.bytes_used: fast, slightly stale bytesize
* account.bytes_used_exact / board.bytes_used_exact: real-time bytesize
* Storage::Entry: audit trail for debugging and point-in-time queries
Process variants synchronously on attachment to close the window between
image upload and variant availability, guaranteeing that we won't have
lazy variant processing attempts in GET requests.
Tradeoff is that we do variant processing in upload requests, which is
actually desirable. We're working with images that should take
milliseconds to resize given that we'll already have the file on hand.
References https://github.com/rails/rails/pull/51951
to avoid implicitly creating a variant during a GET to a read replica,
which will raise an exception and cause the image to fail to load.
This feels pretty brittle, to be honest, and we'll need to be careful
in the future about creating and using new variants; we will probably
need to backfill the new variant for existing images because we can't
guarantee that they will be created implicitly/lazily if the GET lands
on the replica.
ref: https://fizzy.37signals.com/5986089/collections/7/cards/942
Rosa Gutierrez