Rebuilds search records for every published Card and every Comment on a
published Card. Intended for manual invocation after a search-index loss
event — not a recurring job.
Uses ActiveJob::Continuable so the work survives worker restarts and
resumes from the last-processed id. The job is idempotent.
Removes script/maintenance/reindex_stale_search_records.rb, which was
written against a different hypothesis (a date-cutoff backfill) and is
superseded by this unconditional repair job.
The data import path (Account::DataTransfer) uses insert_all! which
bypasses ActiveRecord callbacks, so imported cards and comments never
get search records created and are invisible to search and card-table
filtering.
This script reindexes all published cards and comments created before
a cutoff date (default 2026-04-01). The reindex method is an upsert
so already-indexed records get a harmless update.
Usage:
bin/rails runner script/maintenance/reindex_stale_search_records.rb
bin/rails runner script/maintenance/reindex_stale_search_records.rb 2026-01-01
One-time script to clean up any search records that were indexed
before the searchable? guard was added.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Storage: harden reconcile for concurrent writes and fix board transfer
Reconcile now uses two-cursor approach: captures cursor before and after
the storage scan, aborting if they differ (entries added during scan).
Job retries 3x with 1-minute waits and limits concurrency to 1 per owner.
Board transfer now correctly moves storage for card description embeds
and comment embeds, not just direct attachments. Uses batched queries
to handle cards with thousands of comments efficiently.
Also fixes N+1 queries in attachment grouping via lookup maps.
* Storage: fix per-attachment reconcile and enforce no blob reuse
The storage ledger tracks per-attachment (not per-blob) as a business
abstraction for quotas. This fixes reconcile to match that model and
adds enforcement to prevent blob reuse in tracked contexts.
* Reconcile now uses joins(:blob).sum() for per-attachment counting
* New validation prevents reusing blobs across tracked attachments
* Race-safe storage_total creation with create_or_find_by
* Job efficiency: skip find_by when object already available
* Backfill script checks per-attachment, not just per-blob
An append-only storage ledger replaces deadlock-prone synchronous
counter updates and drift-prone async updates.
All content storage (card images, card/comment/board description embeds)
is tracked. Account exports (which expire) and avatars are not tracked.
Storage is accounted for by Account and Board. Both consume the same
event stream independently: no bubble-up storage bumps triggering
deadlocks due to lock sequencing. Each calculates its own total from
the underlying ledger with independent cursors and materialization.
* Storage::Entry: Append-only ledger recording attach/detach/transfer
events with delta bytes. Single event stream indexed for both
Account and Board cursor queries.
* Storage::Total: Polymorphic snapshot cache with cursor (last_entry_id)
tracking which entries have been materialized.
* Storage::Totaled concern: Provides bytes_used (fast snapshot) and
bytes_used_exact (snapshot + pending) query modes, plus
materialize_storage! to roll up pending entries.
* Storage::Tracked concern: For models owning attachments (Card,
Comment, Board). Provides board_for_storage_tracking for models
where board is determined differently (Board returns self).
Handles board transfers by recording transfer_out/transfer_in entries.
* Storage::AttachmentTracking: Hooks ActiveStorage::Attachment lifecycle
to record attach/detach entries. Handles ActionText::RichText embeds
by traversing to the actual model. Snapshots context in before_destroy
to handle cascading deletes where parent record may be gone by
after_destroy_commit.
* MaterializeJob: Rolls up pending entries into snapshot. Concurrency
limited per owner to prevent duplicate work.
* ReconcileJob: On-demand reconciliation against actual attachment
storage for support/debugging. Compares ledger total to real bytes
from card images, card embeds, comment embeds, and board embeds.
Usage:
* account.bytes_used / board.bytes_used: fast, slightly stale bytesize
* account.bytes_used_exact / board.bytes_used_exact: real-time bytesize
* Storage::Entry: audit trail for debugging and point-in-time queries
Identifies users who have demonstrated authentication by looking for
evidence of real activity: owners, content creators (cards, comments,
boards, events), action takers (assigners, closers, postponers,
reactors, pinners, filter creators), board accessors, export requesters,
push subscribers, setup completers, and users with active sessions.
Run this script after deploying the verified_at column to backfill
existing users before enabling the notification guard.
Co-Authored-By: Claude <noreply@anthropic.com>
Instead of writer pinning, we'll track the last transaction ID of each
write in the session. Then on each read we'll wait for the replica to
report that this transaction is available.
If it doesn't become available within a reasonable timeout, we'll
proceed anyway, and accept the possibility of a stale read.
The hope here is that most of the time, the replica is caught up in the
time between a write request and the following read request. If it's
not, we now have a little tolerance to wait for it, which hopefully
proves enough to stale reads are not encountered in normal use.
We also disable the writer affinity opt-out mechanism that we had
before, since we will no longer be using writer affinity at the load
balancer.
This adds an initial configuration for staging. Note that we are doing
this ahead of having the full infrastructure in all 3 DCs. So this
this will result in some cross-DC writes for now (e.g. we have a single cache in
IAD). We'll correct this as the infrastructure becomes available.
We'll also run jobs on every app server, until we split them out to
separate instances.
Schema:
- add account_id to tables it was missing from
- make account_id a required column everywhere
- add [account_id] indexes, or add `account_id` to existing indices
Models:
- add `belongs_to :account` to all models (default to using a domain
model's account whenever possible)
- add account_id in all the necessary fixtures
- add account_id to insert_all hashes
- pass account_id to a few initialize calls
Miscellaneous:
- update the import script to set account_id
Note that I'm not adding account_id to the join tables primarily
because I couldn't think of an easy way to populate it without making
it a full Join model, and that was more work than I have time to take
on right now.
- Switch to binary 16 for UUID keys
- Remove AccountScopedRecord base class, all model use binary uuids now
- Fix the search sql to serialize uuids properly
- Patch the MySQL schema dumper to output binary lengths
Mike Dalessio