Jeremy Daer
761d0b4a76
An append-only storage ledger replaces deadlock-prone synchronous counter updates and drift-prone async updates. All content storage (card images, card/comment/board description embeds) is tracked. Account exports (which expire) and avatars are not tracked. Storage is accounted for by Account and Board. Both consume the same event stream independently: no bubble-up storage bumps triggering deadlocks due to lock sequencing. Each calculates its own total from the underlying ledger with independent cursors and materialization. * Storage::Entry: Append-only ledger recording attach/detach/transfer events with delta bytes. Single event stream indexed for both Account and Board cursor queries. * Storage::Total: Polymorphic snapshot cache with cursor (last_entry_id) tracking which entries have been materialized. * Storage::Totaled concern: Provides bytes_used (fast snapshot) and bytes_used_exact (snapshot + pending) query modes, plus materialize_storage! to roll up pending entries. * Storage::Tracked concern: For models owning attachments (Card, Comment, Board). Provides board_for_storage_tracking for models where board is determined differently (Board returns self). Handles board transfers by recording transfer_out/transfer_in entries. * Storage::AttachmentTracking: Hooks ActiveStorage::Attachment lifecycle to record attach/detach entries. Handles ActionText::RichText embeds by traversing to the actual model. Snapshots context in before_destroy to handle cascading deletes where parent record may be gone by after_destroy_commit. * MaterializeJob: Rolls up pending entries into snapshot. Concurrency limited per owner to prevent duplicate work. * ReconcileJob: On-demand reconciliation against actual attachment storage for support/debugging. Compares ledger total to real bytes from card images, card embeds, comment embeds, and board embeds. Usage: * account.bytes_used / board.bytes_used: fast, slightly stale bytesize * account.bytes_used_exact / board.bytes_used_exact: real-time bytesize * Storage::Entry: audit trail for debugging and point-in-time queries
71 lines
2.2 KiB
Ruby
71 lines
2.2 KiB
Ruby
module Storage::Totaled
|
|
extend ActiveSupport::Concern
|
|
|
|
included do
|
|
has_one :storage_total, as: :owner, class_name: "Storage::Total", dependent: :destroy
|
|
has_many :storage_entries, class_name: "Storage::Entry", foreign_key: foreign_key_for_storage
|
|
end
|
|
|
|
class_methods do
|
|
def foreign_key_for_storage
|
|
"#{model_name.singular}_id"
|
|
end
|
|
end
|
|
|
|
# Fast: materialized snapshot (may be slightly stale)
|
|
def bytes_used
|
|
storage_total&.bytes_stored || 0
|
|
end
|
|
|
|
# Exact: snapshot + pending entries
|
|
def bytes_used_exact
|
|
(storage_total || create_storage_total!).current_usage
|
|
end
|
|
|
|
def materialize_storage_later
|
|
Storage::MaterializeJob.perform_later(self)
|
|
end
|
|
|
|
# Materialize all pending entries into snapshot
|
|
def materialize_storage
|
|
total = storage_total || create_storage_total!
|
|
|
|
total.with_lock do
|
|
latest_entry_id = storage_entries.maximum(:id)
|
|
|
|
if latest_entry_id && total.last_entry_id != latest_entry_id
|
|
scope = storage_entries.where(id: ..latest_entry_id)
|
|
scope = scope.where.not(id: ..total.last_entry_id) if total.last_entry_id
|
|
delta_sum = scope.sum(:delta)
|
|
|
|
total.update! bytes_stored: total.bytes_stored + delta_sum, last_entry_id: latest_entry_id
|
|
end
|
|
end
|
|
end
|
|
|
|
# Reconcile ledger against actual attachment storage.
|
|
# Uses cursor to ensure consistency: captures max entry ID first, then calculates
|
|
# real bytes, then sums only entries up to that cursor. Concurrent uploads during
|
|
# calculation will have entries with IDs beyond the cursor, avoiding double-count.
|
|
def reconcile_storage
|
|
max_entry_id = storage_entries.maximum(:id)
|
|
real_bytes = calculate_real_storage_bytes
|
|
ledger_bytes = max_entry_id ? storage_entries.where(id: ..max_entry_id).sum(:delta) : 0
|
|
diff = real_bytes - ledger_bytes
|
|
|
|
if diff.nonzero?
|
|
Storage::Entry.record \
|
|
account: is_a?(Account) ? self : account,
|
|
board: is_a?(Board) ? self : nil,
|
|
recordable: nil,
|
|
delta: diff,
|
|
operation: "reconcile"
|
|
end
|
|
end
|
|
|
|
private
|
|
def calculate_real_storage_bytes
|
|
raise NotImplementedError, "Subclass must implement calculate_real_storage_bytes"
|
|
end
|
|
end
|