* Storage: harden reconcile for concurrent writes and fix board transfer
Reconcile now uses two-cursor approach: captures cursor before and after
the storage scan, aborting if they differ (entries added during scan).
Job retries 3x with 1-minute waits and limits concurrency to 1 per owner.
Board transfer now correctly moves storage for card description embeds
and comment embeds, not just direct attachments. Uses batched queries
to handle cards with thousands of comments efficiently.
Also fixes N+1 queries in attachment grouping via lookup maps.
* Storage: fix per-attachment reconcile and enforce no blob reuse
The storage ledger tracks per-attachment (not per-blob) as a business
abstraction for quotas. This fixes reconcile to match that model and
adds enforcement to prevent blob reuse in tracked contexts.
* Reconcile now uses joins(:blob).sum() for per-attachment counting
* New validation prevents reusing blobs across tracked attachments
* Race-safe storage_total creation with create_or_find_by
* Job efficiency: skip find_by when object already available
* Backfill script checks per-attachment, not just per-blob
An append-only storage ledger replaces deadlock-prone synchronous
counter updates and drift-prone async updates.
All content storage (card images, card/comment/board description embeds)
is tracked. Account exports (which expire) and avatars are not tracked.
Storage is accounted for by Account and Board. Both consume the same
event stream independently: no bubble-up storage bumps triggering
deadlocks due to lock sequencing. Each calculates its own total from
the underlying ledger with independent cursors and materialization.
* Storage::Entry: Append-only ledger recording attach/detach/transfer
events with delta bytes. Single event stream indexed for both
Account and Board cursor queries.
* Storage::Total: Polymorphic snapshot cache with cursor (last_entry_id)
tracking which entries have been materialized.
* Storage::Totaled concern: Provides bytes_used (fast snapshot) and
bytes_used_exact (snapshot + pending) query modes, plus
materialize_storage! to roll up pending entries.
* Storage::Tracked concern: For models owning attachments (Card,
Comment, Board). Provides board_for_storage_tracking for models
where board is determined differently (Board returns self).
Handles board transfers by recording transfer_out/transfer_in entries.
* Storage::AttachmentTracking: Hooks ActiveStorage::Attachment lifecycle
to record attach/detach entries. Handles ActionText::RichText embeds
by traversing to the actual model. Snapshots context in before_destroy
to handle cascading deletes where parent record may be gone by
after_destroy_commit.
* MaterializeJob: Rolls up pending entries into snapshot. Concurrency
limited per owner to prevent duplicate work.
* ReconcileJob: On-demand reconciliation against actual attachment
storage for support/debugging. Compares ledger total to real bytes
from card images, card embeds, comment embeds, and board embeds.
Usage:
* account.bytes_used / board.bytes_used: fast, slightly stale bytesize
* account.bytes_used_exact / board.bytes_used_exact: real-time bytesize
* Storage::Entry: audit trail for debugging and point-in-time queries
Process variants synchronously on attachment to close the window between
image upload and variant availability, guaranteeing that we won't have
lazy variant processing attempts in GET requests.
Tradeoff is that we do variant processing in upload requests, which is
actually desirable. We're working with images that should take
milliseconds to resize given that we'll already have the file on hand.
References https://github.com/rails/rails/pull/51951
Aligns with Rails conventions for organizing concerns in a dedicated
concerns directory alongside existing concerns like Attachments,
Mentions, Searchable, etc.
Patch the sqlite adapter to add CHECK constraints for string and text
column limits. We'll do them inline, so that any column changes
automatically update the constraints.
It doesn't actually work, and even if we could make it work reliably
we are better off if the records always know to go to the right shard.
It does make the interface a bit more complicated as we need to select
the right shard class with `for(account_id)`.
Instead we'll compute the table name dynamically based on
Current.account where needed. Also we'll prevent searchable records
from being saved if Current.account is not set, otherwise the after
commit callbacks will fail.
Add an account key field to improve search performance. This field
allows us to filter the records by account directly in the fulltext
index, so we only need to examine rows belonging to the relevant account.
Lean on ActiveRecord models for searching and strip out the raw SQL.
Replaces the search_index_* tables with sharded search_records_* tables
as that allows us to use a Search::Record model name.
A Class is dynamically created for each record table shard so that we
and we can access it via the Search::Record.for_account(account_id)
method.
- Switch to binary 16 for UUID keys
- Remove AccountScopedRecord base class, all model use binary uuids now
- Fix the search sql to serialize uuids properly
- Patch the MySQL schema dumper to output binary lengths
Create search_index_0 to search_index_15 tables and shard each index by
account id. MySQL has no ability to pre-filter fulltext indexes by
another field so this is the best bet for improving performance.
Each fulltest index internally creates 11 sub tables (see
https://dev.mysql.com/doc/refman/8.4/en/innodb-fulltext-index.html) so
actually we have 192 tables in total here.
The search_index table name is generated dynamically based on the
account_id.
Add a single search_index table for full-text search of cards and
comments.
For the search there is a full-text index on the title and content
columns. The board_ids is also included in the table and accessible
board ids are pre-loaded and included in the search query. This allows
us to filter out inaccessible records before joining with other tables.
Right now the search is just using boolean search. This would give us
a bunch of syntax options
(see https://dev.mysql.com/doc/refman/8.4/en/fulltext-boolean.html)
except the search query filters those out.
I've removed the searchable_by method for now - everything is built
on the assumption that there's a single search index table and all data
must fit into it.
Queries are written in SQL, we don't have a SearchIndex ActiveRecord
model. That's because we'll likely want to shard the table and it will
be simpler to just keep with the raw SQL for that.
There's no stemming, highlighting or snippet extraction yet - we are
dumping the full description in the search results.
Data can be reindexed with the search:reindex rake task.
This wasn't working when creation and publishing happened on separated instances
of the same record, which is what happens when you edit a draft and then you
publish it
https://app.fizzy.do/5986089/cards/2835/edit
* main: (28 commits)
Clean up the /scripts directory
Prevent recursion when invoking AI
Bump sqlite3 from 2.7.2 to 2.7.3
Port over Jeremy's fix for slow uploads
.gitignore log files and portfolio's local storage
Drop Current.account and just use Account.sole
Use the utf8-patched mysql driver
Call blob.preview for previewable attachments
Make sure the image variants we use are preprocessed
Add /user command
Make sure it doesn't infer custom abbreviations when matching user names
Script to load prod db in development
Update Signup to use the queenbee_id for the tenant name
Bump cache version
Make sure it uses latest bundler to prevent error due to mismatch
Invalidate cache
Update ruby
Update bundler to prevent error on lower version
Update bundler to prevent error on lower version
Resolve stages
...
to avoid implicitly creating a variant during a GET to a read replica,
which will raise an exception and cause the image to fail to load.
This feels pretty brittle, to be honest, and we'll need to be careful
in the future about creating and using new variants; we will probably
need to backfill the new variant for existing images because we can't
guarantee that they will be created implicitly/lazily if the GET lands
on the replica.
ref: https://fizzy.37signals.com/5986089/collections/7/cards/942
Jeremy Daer