The stemmer was concatenating tokens across hyphens (e.g. "BC3-IOS-1D8B"
→ "bc3ios1d8b") while the query sanitizer split on them, causing MySQL
MATCH AGAINST to never find the indexed token. Replace non-word chars
with spaces instead of stripping them so indexing and querying tokenize
consistently.
Requires search:reindex after deploy to rebuild existing search records.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split each CJK character into individual tokens for FTS5 indexing
- Add stem_content callback to SQLite adapter to pre-process content
- Stem search queries before matching against FTS5 index
- Update stemmer tests to reflect character-splitting behavior
This enables CJK search on SQLite installations where the FTS5
tokenizer cannot natively segment CJK text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The search functionality was silently dropping CJK characters because:
1. Query sanitization used `\w` which only matches ASCII word characters
2. Stemmer split by whitespace, which doesn't work for CJK languages
3. Highlighter used `\b` word boundaries that don't apply to CJK
This commit fixes all three issues:
- Query: Use `\p{L}\p{N}` (Unicode letters/numbers) instead of `\w`
- Stemmer: Preserve CJK characters as-is without stemming, since CJK
languages don't have stemming rules like English
- Highlighter: Skip word boundary matching for CJK terms
Also extracts `CJK_PATTERN` to `Search` module to avoid duplication.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Patch load_schmema! to set the default value for UUID primary keys. This
removes the need to patch ApplicationRecord + Rails models individually.
It also means we no longer need to patch the default in for the integer
primary key in Search::Record::SQLite.
It doesn't actually work, and even if we could make it work reliably
we are better off if the records always know to go to the right shard.
It does make the interface a bit more complicated as we need to select
the right shard class with `for(account_id)`.
Instead we'll compute the table name dynamically based on
Current.account where needed. Also we'll prevent searchable records
from being saved if Current.account is not set, otherwise the after
commit callbacks will fail.
Add an account key field to improve search performance. This field
allows us to filter the records by account directly in the fulltext
index, so we only need to examine rows belonging to the relevant account.
Lean on ActiveRecord models for searching and strip out the raw SQL.
Replaces the search_index_* tables with sharded search_records_* tables
as that allows us to use a Search::Record model name.
A Class is dynamically created for each record table shard so that we
and we can access it via the Search::Record.for_account(account_id)
method.
Schema:
- add account_id to tables it was missing from
- make account_id a required column everywhere
- add [account_id] indexes, or add `account_id` to existing indices
Models:
- add `belongs_to :account` to all models (default to using a domain
model's account whenever possible)
- add account_id in all the necessary fixtures
- add account_id to insert_all hashes
- pass account_id to a few initialize calls
Miscellaneous:
- update the import script to set account_id
Note that I'm not adding account_id to the join tables primarily
because I couldn't think of an easy way to populate it without making
it a full Join model, and that was more work than I have time to take
on right now.
- Switch to binary 16 for UUID keys
- Remove AccountScopedRecord base class, all model use binary uuids now
- Fix the search sql to serialize uuids properly
- Patch the MySQL schema dumper to output binary lengths
Fernando Olivares