Storage: harden reconcile for concurrent writes and fix board transfer (#2096)

* Storage: harden reconcile for concurrent writes and fix board transfer

Reconcile now uses two-cursor approach: captures cursor before and after
the storage scan, aborting if they differ (entries added during scan).
Job retries 3x with 1-minute waits and limits concurrency to 1 per owner.

Board transfer now correctly moves storage for card description embeds
and comment embeds, not just direct attachments. Uses batched queries
to handle cards with thousands of comments efficiently.

Also fixes N+1 queries in attachment grouping via lookup maps.

* Storage: fix per-attachment reconcile and enforce no blob reuse

The storage ledger tracks per-attachment (not per-blob) as a business
abstraction for quotas. This fixes reconcile to match that model and
adds enforcement to prevent blob reuse in tracked contexts.

* Reconcile now uses joins(:blob).sum() for per-attachment counting
* New validation prevents reusing blobs across tracked attachments
* Race-safe storage_total creation with create_or_find_by
* Job efficiency: skip find_by when object already available
* Backfill script checks per-attachment, not just per-blob
This commit is contained in:
Jeremy Daer
2025-12-19 13:07:32 -08:00
committed by GitHub
parent 3a41ac3e23
commit bd6c1cf34f
15 changed files with 669 additions and 85 deletions
+19 -2
View File
@@ -8,7 +8,21 @@
# Run via Kamal:
# kamal app exec -d <stage> -p --reuse "bin/rails runner script/migrations/backfill-storage-ledger.rb"
#
# Safe to re-run: skips attachments that already have entries (by blob_id).
# Safe to re-run: skips attachments that already have entries (by blob_id + recordable).
#
# IMPORTANT: Before running in production, verify zero historic blob reuse in tracked contexts:
#
# ActiveStorage::Attachment
# .joins(:blob)
# .where(record_type: Storage::TRACKED_RECORD_TYPES)
# .where.not(active_storage_blobs: { account_id: Storage::TEMPLATE_ACCOUNT_ID })
# .select(:blob_id)
# .group(:blob_id)
# .having("COUNT(*) > 1")
# .count
# # Should return empty hash if no reuse exists in tracked contexts
#
# If reuse exists (excluding template blobs), fix the data first.
class BackfillStorageLedger
def run
puts "Backfilling storage entries…"
@@ -26,7 +40,10 @@ class BackfillStorageLedger
ActiveStorage::Attachment.includes(:blob).find_each do |attachment|
record = attachment.record.try(:storage_tracked_record)
if record.nil? || Storage::Entry.exists?(blob_id: attachment.blob_id)
# Backfill creates one entry PER ATTACHMENT (not per blob) to match the ledger model.
# Storage tracking is a business abstraction at the attachment level.
# IMPORTANT: This assumes no historic blob reuse. Run pre-check query above first.
if record.nil? || Storage::Entry.exists?(blob_id: attachment.blob_id, recordable: record)
skipped += 1
next
end