Jeremy Daer
bd6c1cf34f
* Storage: harden reconcile for concurrent writes and fix board transfer Reconcile now uses two-cursor approach: captures cursor before and after the storage scan, aborting if they differ (entries added during scan). Job retries 3x with 1-minute waits and limits concurrency to 1 per owner. Board transfer now correctly moves storage for card description embeds and comment embeds, not just direct attachments. Uses batched queries to handle cards with thousands of comments efficiently. Also fixes N+1 queries in attachment grouping via lookup maps. * Storage: fix per-attachment reconcile and enforce no blob reuse The storage ledger tracks per-attachment (not per-blob) as a business abstraction for quotas. This fixes reconcile to match that model and adds enforcement to prevent blob reuse in tracked contexts. * Reconcile now uses joins(:blob).sum() for per-attachment counting * New validation prevents reusing blobs across tracked attachments * Race-safe storage_total creation with create_or_find_by * Job efficiency: skip find_by when object already available * Backfill script checks per-attachment, not just per-blob
73 lines
2.3 KiB
Ruby
73 lines
2.3 KiB
Ruby
module Board::Storage
|
|
extend ActiveSupport::Concern
|
|
include Storage::Totaled
|
|
|
|
# Board's own embeds (public_description) count toward itself
|
|
def board_for_storage_tracking
|
|
self
|
|
end
|
|
|
|
private
|
|
BATCH_SIZE = 1000
|
|
|
|
# Calculate actual storage by summing blob sizes.
|
|
#
|
|
# Storage tracking is a business abstraction - we count what users upload.
|
|
# Original upload bytes only; variants/previews/derivatives excluded.
|
|
# Physical storage optimizations (deduplication, compression) don't affect quotas.
|
|
def calculate_real_storage_bytes
|
|
@card_ids = nil # Clear memoization for fresh calculation
|
|
card_image_bytes + card_embed_bytes + comment_embed_bytes + board_embed_bytes
|
|
end
|
|
|
|
def card_ids
|
|
@card_ids ||= cards.pluck(:id)
|
|
end
|
|
|
|
def card_image_bytes
|
|
sum_blob_bytes_in_batches \
|
|
ActiveStorage::Attachment.where(record_type: "Card", name: "image"),
|
|
card_ids
|
|
end
|
|
|
|
def card_embed_bytes
|
|
sum_embed_bytes_for "Card", card_ids
|
|
end
|
|
|
|
def comment_embed_bytes
|
|
card_ids.each_slice(BATCH_SIZE).sum do |batch|
|
|
sum_embed_bytes_for "Comment", Comment.where(card_id: batch).pluck(:id)
|
|
end
|
|
end
|
|
|
|
def board_embed_bytes
|
|
sum_embed_bytes_for "Board", [ id ]
|
|
end
|
|
|
|
def sum_embed_bytes_for(record_type, record_ids)
|
|
rich_text_ids = ActionText::RichText \
|
|
.where(record_type: record_type, record_id: record_ids)
|
|
.pluck(:id)
|
|
|
|
sum_blob_bytes_in_batches \
|
|
ActiveStorage::Attachment.where(record_type: "ActionText::RichText", name: "embeds"),
|
|
rich_text_ids
|
|
end
|
|
|
|
def sum_blob_bytes_in_batches(base_scope, record_ids)
|
|
# Count per-attachment to match ledger model.
|
|
# Same blob attached 3 times = 3x bytes (business abstraction, not physical storage).
|
|
#
|
|
# Do NOT remove the join thinking it's a performance optimization - it's required
|
|
# for correct per-attachment counting. We keep ActiveStorage/ActionText in the same
|
|
# database (realm/geo partitioning, not functionality partitioning), so cross-table
|
|
# joins are fine.
|
|
record_ids.each_slice(BATCH_SIZE).sum do |batch_ids|
|
|
base_scope
|
|
.where(record_id: batch_ids)
|
|
.joins(:blob)
|
|
.sum("active_storage_blobs.byte_size")
|
|
end
|
|
end
|
|
end
|