Follow-up to #2550 which added a created_at index and increased cleanup
frequency. The delete itself was still unbatched and vulnerable whenever
a backlog accumulates (first run after deploy, clock skew, etc).
Delete in small batches with a pause between each to let other
transactions through, following the SolidQueue pattern.
This addresses a DoS vulnerability where the response might be massive
leading to OOM errors, as the response is read in full in memory by
default.
To prevent this, we need to read the body in chunks, checking the
size of the chunks we've read and raising if we go over a certain limit.
I've set the limit to 100 KB because the responses to these requests
should be fairly small or even empty, and we only care about the status
code in the end.
Schema:
- add account_id to tables it was missing from
- make account_id a required column everywhere
- add [account_id] indexes, or add `account_id` to existing indices
Models:
- add `belongs_to :account` to all models (default to using a domain
model's account whenever possible)
- add account_id in all the necessary fixtures
- add account_id to insert_all hashes
- pass account_id to a few initialize calls
Miscellaneous:
- update the import script to set account_id
Note that I'm not adding account_id to the join tables primarily
because I couldn't think of an easy way to populate it without making
it a full Join model, and that was more work than I have time to take
on right now.
- Switch to binary 16 for UUID keys
- Remove AccountScopedRecord base class, all model use binary uuids now
- Fix the search sql to serialize uuids properly
- Patch the MySQL schema dumper to output binary lengths
Jeremy Daer