Add an account key field to improve search performance. This field
allows us to filter the records by account directly in the fulltext
index, so we only need to examine rows belonging to the relevant account.
The check-then-act pattern in `register_activity_spike` has been
replaced with `find_or_create_by!` to eliminate the race condition
that could lead to creating multiple activity spikes for a card. To
support this change, the `card_id` index on `cards_activity_spikes`
has been made unique.
ref: https://app.fizzy.do/5986089/cards/3063
These have been a contributing cause to some deadlocks, but also this
makes some operational tasks harder (data loading, data truncation)
and has some performance impact on insert/update/delete operations.
The team consensus is that we aren't relying on them, and so we're
comfortable with the referential integrity risk.
Lean on ActiveRecord models for searching and strip out the raw SQL.
Replaces the search_index_* tables with sharded search_records_* tables
as that allows us to use a Search::Record model name.
A Class is dynamically created for each record table shard so that we
and we can access it via the Search::Record.for_account(account_id)
method.
Schema:
- add account_id to tables it was missing from
- make account_id a required column everywhere
- add [account_id] indexes, or add `account_id` to existing indices
Models:
- add `belongs_to :account` to all models (default to using a domain
model's account whenever possible)
- add account_id in all the necessary fixtures
- add account_id to insert_all hashes
- pass account_id to a few initialize calls
Miscellaneous:
- update the import script to set account_id
Note that I'm not adding account_id to the join tables primarily
because I couldn't think of an easy way to populate it without making
it a full Join model, and that was more work than I have time to take
on right now.
This matches the format in the migration files and makes the type_to_sql
override redundant.
The only wart we are left with is that since there are no true UUID
types in MySQL we'll have to assume that binary(16) columns are UUIDs.
In practice this is probably fine as the MySQL adapter doesn't map any
other types to binary.
- Switch to binary 16 for UUID keys
- Remove AccountScopedRecord base class, all model use binary uuids now
- Fix the search sql to serialize uuids properly
- Patch the MySQL schema dumper to output binary lengths
Create search_index_0 to search_index_15 tables and shard each index by
account id. MySQL has no ability to pre-filter fulltext indexes by
another field so this is the best bet for improving performance.
Each fulltest index internally creates 11 sub tables (see
https://dev.mysql.com/doc/refman/8.4/en/innodb-fulltext-index.html) so
actually we have 192 tables in total here.
The search_index table name is generated dynamically based on the
account_id.
Add a single search_index table for full-text search of cards and
comments.
For the search there is a full-text index on the title and content
columns. The board_ids is also included in the table and accessible
board ids are pre-loaded and included in the search query. This allows
us to filter out inaccessible records before joining with other tables.
Right now the search is just using boolean search. This would give us
a bunch of syntax options
(see https://dev.mysql.com/doc/refman/8.4/en/fulltext-boolean.html)
except the search query filters those out.
I've removed the searchable_by method for now - everything is built
on the assumption that there's a single search index table and all data
must fit into it.
Queries are written in SQL, we don't have a SearchIndex ActiveRecord
model. That's because we'll likely want to shard the table and it will
be simpler to just keep with the raw SQL for that.
There's no stemming, highlighting or snippet extraction yet - we are
dumping the full description in the search results.
Data can be reindexed with the search:reindex rake task.
Once the sqlite 'vec0' module was removed from the codebase in
38a7a144 (following the table removal in 875a298f), this migration
became unrunnable. So to make sure we can reconstruct the schema if
necessary by running all the migrations, I'm replacing it with the
creation of an empty table.
Removes the creating status completely as well as the abandoned cards system. It will
always resume drafts if they exist, for a given collection and user.
https://app.box-car.com/5986089/cards/2489
Stanko K.R.