Speedy, auditable, deadlock-resistant storage tracking (#2026)
An append-only storage ledger replaces deadlock-prone synchronous counter updates and drift-prone async updates. All content storage (card images, card/comment/board description embeds) is tracked. Account exports (which expire) and avatars are not tracked. Storage is accounted for by Account and Board. Both consume the same event stream independently: no bubble-up storage bumps triggering deadlocks due to lock sequencing. Each calculates its own total from the underlying ledger with independent cursors and materialization. * Storage::Entry: Append-only ledger recording attach/detach/transfer events with delta bytes. Single event stream indexed for both Account and Board cursor queries. * Storage::Total: Polymorphic snapshot cache with cursor (last_entry_id) tracking which entries have been materialized. * Storage::Totaled concern: Provides bytes_used (fast snapshot) and bytes_used_exact (snapshot + pending) query modes, plus materialize_storage! to roll up pending entries. * Storage::Tracked concern: For models owning attachments (Card, Comment, Board). Provides board_for_storage_tracking for models where board is determined differently (Board returns self). Handles board transfers by recording transfer_out/transfer_in entries. * Storage::AttachmentTracking: Hooks ActiveStorage::Attachment lifecycle to record attach/detach entries. Handles ActionText::RichText embeds by traversing to the actual model. Snapshots context in before_destroy to handle cascading deletes where parent record may be gone by after_destroy_commit. * MaterializeJob: Rolls up pending entries into snapshot. Concurrency limited per owner to prevent duplicate work. * ReconcileJob: On-demand reconciliation against actual attachment storage for support/debugging. Compares ledger total to real bytes from card images, card embeds, comment embeds, and board embeds. Usage: * account.bytes_used / board.bytes_used: fast, slightly stale bytesize * account.bytes_used_exact / board.bytes_used_exact: real-time bytesize * Storage::Entry: audit trail for debugging and point-in-time queries
This commit is contained in:
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env ruby
|
||||
|
||||
# Backfill storage ledger with attach entries for all existing attachments.
|
||||
#
|
||||
# Run locally:
|
||||
# bin/rails runner script/migrations/backfill-storage-ledger.rb
|
||||
#
|
||||
# Run via Kamal:
|
||||
# kamal app exec -d <stage> -p --reuse "bin/rails runner script/migrations/backfill-storage-ledger.rb"
|
||||
#
|
||||
# Safe to re-run: skips attachments that already have entries (by blob_id).
|
||||
class BackfillStorageLedger
|
||||
def run
|
||||
puts "Backfilling storage entries…"
|
||||
backfill_entries
|
||||
|
||||
puts "\nMaterializing totals…"
|
||||
materialize_totals
|
||||
end
|
||||
|
||||
private
|
||||
def backfill_entries
|
||||
created = 0
|
||||
skipped = 0
|
||||
|
||||
ActiveStorage::Attachment.includes(:blob).find_each do |attachment|
|
||||
record = attachment.record.try(:storage_tracked_record)
|
||||
|
||||
if record.nil? || Storage::Entry.exists?(blob_id: attachment.blob_id)
|
||||
skipped += 1
|
||||
next
|
||||
end
|
||||
|
||||
Storage::Entry.create! \
|
||||
account_id: record.account.id,
|
||||
board_id: record.board_for_storage_tracking&.id,
|
||||
recordable_type: record.class.name,
|
||||
recordable_id: record.id,
|
||||
blob_id: attachment.blob_id,
|
||||
delta: attachment.blob.byte_size,
|
||||
operation: "attach"
|
||||
created += 1
|
||||
|
||||
print "." if created % 100 == 0
|
||||
end
|
||||
|
||||
puts "\n\nBackfill complete!"
|
||||
puts " Entries created: #{created}"
|
||||
puts " Attachments skipped: #{skipped}"
|
||||
end
|
||||
|
||||
def materialize_totals
|
||||
boards_materialized = 0
|
||||
accounts_materialized = 0
|
||||
|
||||
Board.find_each do |board|
|
||||
board.materialize_storage
|
||||
boards_materialized += 1
|
||||
print "." if boards_materialized % 100 == 0
|
||||
end
|
||||
|
||||
Account.find_each do |account|
|
||||
account.materialize_storage
|
||||
accounts_materialized += 1
|
||||
end
|
||||
|
||||
puts "\n\nMaterialization complete!"
|
||||
puts " Boards: #{boards_materialized}"
|
||||
puts " Accounts: #{accounts_materialized}"
|
||||
end
|
||||
end
|
||||
|
||||
BackfillStorageLedger.new.run
|
||||
Reference in New Issue
Block a user