LU
2a4fbecbda
The search functionality was silently dropping CJK characters because:
1. Query sanitization used `\w` which only matches ASCII word characters
2. Stemmer split by whitespace, which doesn't work for CJK languages
3. Highlighter used `\b` word boundaries that don't apply to CJK
This commit fixes all three issues:
- Query: Use `\p{L}\p{N}` (Unicode letters/numbers) instead of `\w`
- Stemmer: Preserve CJK characters as-is without stemming, since CJK
languages don't have stemming rules like English
- Highlighter: Skip word boundary matching for CJK terms
Also extracts `CJK_PATTERN` to `Search` module to avoid duplication.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
47 lines
943 B
Ruby
47 lines
943 B
Ruby
class Search::Query < ApplicationRecord
|
|
belongs_to :account, default: -> { user&.account || Current.account }
|
|
belongs_to :user, optional: true
|
|
|
|
validates :terms, presence: true
|
|
before_validation :sanitize_terms
|
|
|
|
delegate :to_s, to: :terms
|
|
|
|
class << self
|
|
def wrap(query)
|
|
if query.is_a?(self)
|
|
query
|
|
else
|
|
self.new(terms: query)
|
|
end
|
|
end
|
|
end
|
|
|
|
private
|
|
def sanitize_terms
|
|
self.terms = sanitize(terms)
|
|
end
|
|
|
|
def sanitize(terms)
|
|
if terms.present?
|
|
terms = remove_invalid_search_characters(self.terms)
|
|
terms = remove_unbalanced_quotes(terms)
|
|
terms.presence
|
|
else
|
|
terms
|
|
end
|
|
end
|
|
|
|
def remove_invalid_search_characters(terms)
|
|
terms.gsub(/[^\p{L}\p{N}_"]/, " ")
|
|
end
|
|
|
|
def remove_unbalanced_quotes(terms)
|
|
if terms.count("\"").even?
|
|
terms
|
|
else
|
|
terms.gsub("\"", " ")
|
|
end
|
|
end
|
|
end
|