- Split each CJK character into individual tokens for FTS5 indexing
- Add stem_content callback to SQLite adapter to pre-process content
- Stem search queries before matching against FTS5 index
- Update stemmer tests to reflect character-splitting behavior
This enables CJK search on SQLite installations where the FTS5
tokenizer cannot natively segment CJK text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The search functionality was silently dropping CJK characters because:
1. Query sanitization used `\w` which only matches ASCII word characters
2. Stemmer split by whitespace, which doesn't work for CJK languages
3. Highlighter used `\b` word boundaries that don't apply to CJK
This commit fixes all three issues:
- Query: Use `\p{L}\p{N}` (Unicode letters/numbers) instead of `\w`
- Stemmer: Preserve CJK characters as-is without stemming, since CJK
languages don't have stemming rules like English
- Highlighter: Skip word boundary matching for CJK terms
Also extracts `CJK_PATTERN` to `Search` module to avoid duplication.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The SSRF filter checked ipv4_mapped? but not ipv4_compat?, allowing
addresses like ::169.254.169.254 to bypass the link-local check and
reach cloud metadata endpoints.
Changes:
- Add ipv4_compat? check to block deprecated IPv4-compatible format
- Rename private_address? to blocked_address? (more accurate - method
blocks more than just RFC 1918 private ranges)
- Add IPv6 test coverage for both mapped and compat formats
Both ipv4_mapped and ipv4_compat formats are blocked entirely as
defense-in-depth: DNS never returns these formats, so they only
appear in attack scenarios.
HackerOne: #3481701
- Update `Card::Closeable` to destroy associated "not now" state during close.
- Add new tests to verify closure behavior for different column types, including "not now".
The included SearchTestHelper disables transactional tests and has a specific setup that does not use fixtures.
This test however, ignores the setup and uses fixtures directly by creating comments and publishing cards making test state unpredictable.
By using the records set by the helper instead of fixtures, the state of the system is not polluted.
* Storage: harden reconcile for concurrent writes and fix board transfer
Reconcile now uses two-cursor approach: captures cursor before and after
the storage scan, aborting if they differ (entries added during scan).
Job retries 3x with 1-minute waits and limits concurrency to 1 per owner.
Board transfer now correctly moves storage for card description embeds
and comment embeds, not just direct attachments. Uses batched queries
to handle cards with thousands of comments efficiently.
Also fixes N+1 queries in attachment grouping via lookup maps.
* Storage: fix per-attachment reconcile and enforce no blob reuse
The storage ledger tracks per-attachment (not per-blob) as a business
abstraction for quotas. This fixes reconcile to match that model and
adds enforcement to prevent blob reuse in tracked contexts.
* Reconcile now uses joins(:blob).sum() for per-attachment counting
* New validation prevents reusing blobs across tracked attachments
* Race-safe storage_total creation with create_or_find_by
* Job efficiency: skip find_by when object already available
* Backfill script checks per-attachment, not just per-blob
As the tests for it could lead to confusion where it seems drafted cards
are not accessible to someone with access to the board and with the
direct drafted card URL.
Addresses an issue where User#familiar_name assumed `name` was always
present, potentially raising an exception during view rendering. Now
User validates name presence, and User#familiar_name handles blank
strings without error, in case any existing invalid records exist.
If we don't validate for length, then signups that overflow the database
columns will unnecessarily create and cancel a tenant. Adding a
validation means we can avoid this.
* Add test coverage for with_golden_first scope
Test verifies that the with_golden_first scope correctly orders
golden cards before non-golden cards in query results.
* Refactor golden test: use instance variables, add subordering coverage
Extract @golden and @non_golden fixtures into setup for reuse across
all tests. Simplify with_golden_first test to verify both primary
ordering (golden before non-golden) and subordering preservation.
---------
Co-authored-by: Jeremy Daer <jeremy@37signals.com>
* Allow Card#last_updated_at to be set
This is useful when doing an import from another system. I'm currently
working on a script to import our Github issues into Fizzy.
This is discussed in
https://github.com/basecamp/fizzy/pull/2056#discussion_r2609560246
* Add nil fallback and expand test coverage for last_active_at
Adds a safety fallback to Time.current if created_at is unexpectedly nil
during card creation.
Test coverage to verify:
* last_active_at defaults to created_at when not provided
* last_active_at can be updated via API on existing cards
* import workflow where last_active_at is restored after comments
* publishing doesn't overwrite explicit last_active_at values
---------
Co-authored-by: Jeremy Daer <jeremy@37signals.com>
An append-only storage ledger replaces deadlock-prone synchronous
counter updates and drift-prone async updates.
All content storage (card images, card/comment/board description embeds)
is tracked. Account exports (which expire) and avatars are not tracked.
Storage is accounted for by Account and Board. Both consume the same
event stream independently: no bubble-up storage bumps triggering
deadlocks due to lock sequencing. Each calculates its own total from
the underlying ledger with independent cursors and materialization.
* Storage::Entry: Append-only ledger recording attach/detach/transfer
events with delta bytes. Single event stream indexed for both
Account and Board cursor queries.
* Storage::Total: Polymorphic snapshot cache with cursor (last_entry_id)
tracking which entries have been materialized.
* Storage::Totaled concern: Provides bytes_used (fast snapshot) and
bytes_used_exact (snapshot + pending) query modes, plus
materialize_storage! to roll up pending entries.
* Storage::Tracked concern: For models owning attachments (Card,
Comment, Board). Provides board_for_storage_tracking for models
where board is determined differently (Board returns self).
Handles board transfers by recording transfer_out/transfer_in entries.
* Storage::AttachmentTracking: Hooks ActiveStorage::Attachment lifecycle
to record attach/detach entries. Handles ActionText::RichText embeds
by traversing to the actual model. Snapshots context in before_destroy
to handle cascading deletes where parent record may be gone by
after_destroy_commit.
* MaterializeJob: Rolls up pending entries into snapshot. Concurrency
limited per owner to prevent duplicate work.
* ReconcileJob: On-demand reconciliation against actual attachment
storage for support/debugging. Compares ledger total to real bytes
from card images, card embeds, comment embeds, and board embeds.
Usage:
* account.bytes_used / board.bytes_used: fast, slightly stale bytesize
* account.bytes_used_exact / board.bytes_used_exact: real-time bytesize
* Storage::Entry: audit trail for debugging and point-in-time queries
This addresses a DoS vulnerability where the response might be massive
leading to OOM errors, as the response is read in full in memory by
default.
To prevent this, we need to read the body in chunks, checking the
size of the chunks we've read and raising if we go over a certain limit.
I've set the limit to 100 KB because the responses to these requests
should be fairly small or even empty, and we only care about the status
code in the end.
When ActiveStorage::Record uses `connects_to` for read replica support,
it creates a separate connection pool from ApplicationRecord. This causes
`after_commit` callbacks to fire in non-deterministic order - the
Attachment's `create_variants` callback can fire before the User model's
upload callback completes, resulting in FileNotFoundError.
The fix removes replica connection configuration from ActiveStorage::Record
so it shares the same connection pool as application models, ensuring
proper callback ordering.
Also reverts test workarounds that were added to work around this issue,
since the root cause is now fixed.
See: https://github.com/rails/rails/issues/53694
Adds verified? check to bundling_emails? to prevent notification emails
from being sent to users who have never authenticated. This closes the
spam vector where bad actors could create users for known email
addresses and trigger unwanted notifications by mentioning them.
Co-Authored-By: Claude <noreply@anthropic.com>
User are marked as verified after a join code is redeemed. The user is
redirected to Users::VerificationsController, either:
- after submitting a valid magic link code,
- or immediately after redeeming the join code (if they're already
authenticated with the correct identity)
Account owners are automatically verified when the account is
created (because they have already provided a magic link code at that
point).
This sets up for later commits that will backfill existing users and
require verification before sending notification emails.
Co-Authored-By: Claude <noreply@anthropic.com>
Process variants synchronously on attachment to close the window between
image upload and variant availability, guaranteeing that we won't have
lazy variant processing attempts in GET requests.
Tradeoff is that we do variant processing in upload requests, which is
actually desirable. We're working with images that should take
milliseconds to resize given that we'll already have the file on hand.
References https://github.com/rails/rails/pull/51951
Jorge Manrubia