Add session report (architecture, coverage, ORM vs Postgres+Redis)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
GregandClaude Opus 4.8 committed
1 parent
45659c957e
commit
d2202bda03
1 file changed
+302
@@ -0,0 +1,302 @@
|
|||||||
|
# simply_couch → Couchbase: Session Report
|
||||||
|
|
||||||
|
**For:** Benjamin ter Kuile (bterkuile)
|
||||||
|
**Prepared by:** Greg (via Claude Code / Opus 4.8)
|
||||||
|
**Date:** 2026-06-20
|
||||||
|
|
||||||
|
This document summarizes a working session that started from one request —
|
||||||
|
"compartmentalize simply_couch's data backend so CouchDB could be swapped for
|
||||||
|
Couchbase" — and grew, through your input in the thread, into three concrete
|
||||||
|
artifacts plus an architectural recommendation. Everything below was built and
|
||||||
|
verified against a live database (CouchDB 3.5.2 and Couchbase Community 7.x in
|
||||||
|
local Docker); nothing here is hypothetical.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
Three artifacts, in increasing ambition:
|
||||||
|
|
||||||
|
1. **PR #10 on `bterkuile/simply_couch`** — a behaviour-preserving refactor that
|
||||||
|
turns the leaky single-driver seam into a real, documented adapter contract.
|
||||||
|
Full suite green (453 examples). CouchDB still the default.
|
||||||
|
2. **`simply_couch-couchbase`** — a Couchbase *adapter plugin* for the refactored
|
||||||
|
simply_couch (kept the CouchDB-shaped API). SDK-based, N1QL view translation.
|
||||||
|
3. **`simply_based`** — the *revolutionary* path you asked for: a standalone
|
||||||
|
Couchbase-native ORM. `include SimplyBased::Model`, no CouchDB concepts left.
|
||||||
|
Runs the simply_couch test suite: **403/403 applicable examples green**.
|
||||||
|
|
||||||
|
And a candid recommendation at the end: whether a Couchbase-native ORM is
|
||||||
|
actually the right call versus a standard Postgres + Redis Rails stack.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The core strategy point: Couchbase ≠ CouchDB
|
||||||
|
|
||||||
|
Despite the shared name and lineage (Damien Katz), the two are not API-compatible,
|
||||||
|
and a literal `simply_couchbase_rest` drop-in for `couchrest` cannot exist:
|
||||||
|
|
||||||
|
| Concern | CouchDB (couchrest) | Couchbase |
|
||||||
|
|---|---|---|
|
||||||
|
| Data path | HTTP REST per-document | binary KV via SDK (REST is management-only) |
|
||||||
|
| Query | JS map/reduce **design-doc views** | **N1QL / SQL++** + GSI (CB views deprecated in 7.0) |
|
||||||
|
| Concurrency | MVCC `_rev`, conflict docs | CAS (uint64), no user-facing revisions |
|
||||||
|
| Bulk | `_bulk_docs` | batched KV / subdoc ops |
|
||||||
|
| Attachments | native blob attachments | none — store externally (S3/local) |
|
||||||
|
| "CouchDB interface" | — | only the deprecated Views engine (port 8092 CAPI), or Sync Gateway's replication protocol — neither is a general server API for an ORM |
|
||||||
|
|
||||||
|
So Couchbase is reached the way it's meant to be: the SDK for KV, **N1QL** for
|
||||||
|
queries. Not a REST shim. The pinch point is the view system — but simply_couch
|
||||||
|
builds views from **structured Ruby spec objects**, not ad-hoc JS, so the finder
|
||||||
|
and association queries translate deterministically to N1QL. That's what made the
|
||||||
|
whole thing tractable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Artifact 1 — PR #10: the adapter-contract refactor
|
||||||
|
|
||||||
|
**Link:** https://github.com/bterkuile/simply_couch/pull/10 (open; you were
|
||||||
|
notified on open).
|
||||||
|
|
||||||
|
**What it does (behaviour-preserving — zero functional change for couchrest users):**
|
||||||
|
|
||||||
|
- Adds `SimplyCouch::Adapter` — the backend-neutral contract the model layer
|
||||||
|
depends on (`save_document`, `load_document`, `destroy_document`, `bulk_*`,
|
||||||
|
`view`, `first`, `create_database!`, `drop_database!`), plus an exception
|
||||||
|
contract (drivers translate their errors to `SimplyCouch::Conflict` /
|
||||||
|
`NotFound`).
|
||||||
|
- Moves the existing CouchDB code out of `Model::DatabaseInstance` into
|
||||||
|
`SimplyCouch::Adapters::CouchRest` **verbatim** (a `DatabaseInstance` alias is
|
||||||
|
kept). This is a move, not a rewrite.
|
||||||
|
- **Lazy driver loading:** `require 'simply_couch'` now loads with no driver
|
||||||
|
present; couchrest loads only when its adapter is first used.
|
||||||
|
- Adapter registry: `SimplyCouch.register_adapter(name, class, require_path:)`
|
||||||
|
+ `SimplyCouch.adapter = :name`.
|
||||||
|
- Stops the model/association/view layers reaching through
|
||||||
|
`database.couchrest_database` (5 sites → the neutral `database.bulk_load`).
|
||||||
|
- `SimplyCouch::DocumentCodec` — rebuilds a model from a plain hash via the
|
||||||
|
`ruby_class` tag, so non-couchrest backends can hydrate documents.
|
||||||
|
- Exposes the view-spec `klass`/`options` as the backend-neutral query IR.
|
||||||
|
- Hardens the load guard so a plugin gem can open `module SimplyCouch` for its
|
||||||
|
own namespace before the library loads.
|
||||||
|
- Adds shared `'a simply_couch adapter'` RSpec examples (reusable parity proof).
|
||||||
|
- Version 0.1.2 → 0.2.0; README/CHANGELOG updated.
|
||||||
|
|
||||||
|
**Verification:** 453 examples, 0 failures vs a live CouchDB 3.5.2. No model-facing
|
||||||
|
signature changed. (Two pre-existing flaky DB-reset races predate the PR.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Artifact 2 — `simply_couch-couchbase` (adapter plugin)
|
||||||
|
|
||||||
|
A separate gem that plugs the **refactored** simply_couch into Couchbase while
|
||||||
|
keeping the CouchDB-shaped API. `SimplyCouch::Adapters::Couchbase` on the official
|
||||||
|
`couchbase` SDK: KV for documents, CAS↔`_rev`, and an `N1qlTranslator` that
|
||||||
|
renders the structured `ModelViewSpec` / `DeletedModelViewSpec` (all_documents /
|
||||||
|
find_by / count) to N1QL. Custom/raw JS association views raise
|
||||||
|
`SimplyCouch::UnsupportedView` (documented boundary).
|
||||||
|
|
||||||
|
**Verification:** 20 examples green = 11 N1QL-translator unit tests (no server) +
|
||||||
|
9 live-Couchbase contract examples (the *same* shared examples couchrest passes).
|
||||||
|
|
||||||
|
This is the lower-ambition path: it proves the adapter seam works and gives a
|
||||||
|
working Couchbase backend, but associations were left as an explicit boundary.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Artifact 3 — `simply_based` (the Couchbase-native ORM)
|
||||||
|
|
||||||
|
The path you chose in the thread: not an adapter, a **new gem** that *is* the
|
||||||
|
ORM, Couchbase-native throughout.
|
||||||
|
|
||||||
|
**The one-line swap:**
|
||||||
|
```ruby
|
||||||
|
- include SimplyCouch::Model
|
||||||
|
+ include SimplyBased::Model
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why so little changes:** simply_couch's model layer (properties, type-casting,
|
||||||
|
validations, callbacks, dirty tracking, the association DSL, soft-delete,
|
||||||
|
pagination) is already storage-neutral. SimplyBased keeps all of it and replaces
|
||||||
|
exactly one layer — `SimplyBased::Store` — which speaks Couchbase directly. Every
|
||||||
|
association lookup funnels through a handful of Store methods that emit N1QL.
|
||||||
|
|
||||||
|
**How the CouchDB views became N1QL:**
|
||||||
|
|
||||||
|
- `belongs_to` / `has_many` / `has_one` → scalar foreign-key equality:
|
||||||
|
`WHERE ruby_class = $rc AND user_id = $id`, ordered by `created_at`, GSI auto-
|
||||||
|
created on first use.
|
||||||
|
- `has_and_belongs_to_many` → on the storing side, a **KV multi-get** of the id
|
||||||
|
array (no query at all); on the other side, the two hand-written map/reduce
|
||||||
|
functions become one query:
|
||||||
|
```sql
|
||||||
|
SELECT c.*, META(c).id, META(c).cas
|
||||||
|
FROM `bucket`.`_default`.`_default` AS c
|
||||||
|
WHERE c.ruby_class = $rc
|
||||||
|
AND ANY x IN c.`network_ids` SATISFIES x = $v END
|
||||||
|
```
|
||||||
|
backed by an array index.
|
||||||
|
- **Ancestry / trees** → you were right that it's a nested-set/materialized-path
|
||||||
|
intention. `path_ids` is the full path root→self, so no recursive build:
|
||||||
|
- children = `path_ids[ARRAY_LENGTH-2] = $id`
|
||||||
|
- subtree = `$id IN path_ids` (minus self)
|
||||||
|
- roots = `ARRAY_LENGTH(path_ids) = 1`
|
||||||
|
- `_rev` is mapped onto **CAS**: a record's `_rev` is the stringified CAS of its
|
||||||
|
last write; updates use it for optimistic locking (stale write →
|
||||||
|
`SimplyBased::Conflict`).
|
||||||
|
- Soft-delete-aware `find_by`/`find_all_by`: the `[key, deleted_at] = [val, nil]`
|
||||||
|
convention compares element-wise so a `nil` component matches `MISSING`/`NULL`
|
||||||
|
(non-deleted docs carry no `deleted_at`).
|
||||||
|
- Timestamps are microsecond-precision so same-second creations paginate in
|
||||||
|
creation order deterministically.
|
||||||
|
|
||||||
|
**Replaces Redis too** (same cluster, no second datastore) — 5/5 green:
|
||||||
|
|
||||||
|
- `SimplyBased::Counter` — Couchbase binary increment/decrement (true atomic
|
||||||
|
counters; a drop-in for Redis INCR/DECR).
|
||||||
|
- `SimplyBased::Cache` — an `ActiveSupport::Cache::Store` over KV + native TTL;
|
||||||
|
usable as `Rails.cache` and therefore the ActiveJob / fragment / low-level cache.
|
||||||
|
- `SimplyBased::Lock` — distributed lock via insert-if-absent + TTL auto-expiry +
|
||||||
|
CAS-checked release (only the holder can unlock).
|
||||||
|
|
||||||
|
### Test coverage
|
||||||
|
|
||||||
|
SimplyBased runs the **actual simply_couch test suite** against a live Couchbase.
|
||||||
|
|
||||||
|
- **403 / 403 applicable examples green on a clean run.** One ancestry reparent
|
||||||
|
edge in the namespaced (`by_property`) cascade is intermittently flaky (~50%) —
|
||||||
|
it does several dependent saves and occasionally races; tracked, not yet nailed.
|
||||||
|
- **Green:** CRUD; finders (`find`, `all`, `first`, `find_by_*`, `count_by_*`);
|
||||||
|
`belongs_to` / `has_many` / `has_one` / `has_and_belongs_to_many`; validations;
|
||||||
|
callbacks; dirty tracking; soft-delete (incl. soft-delete-aware finders);
|
||||||
|
pagination; bulk; eager-loading; ancestry/trees; the Redis primitives.
|
||||||
|
- **Excluded by design:**
|
||||||
|
- `:couchdb_only` — native inline attachments, JS-view token-safety,
|
||||||
|
design-document maintenance, raw custom JS views. No Couchbase equivalent
|
||||||
|
(use `has_s3_attached` / `has_local_attached`; N1QL uses bound parameters).
|
||||||
|
- `embedded_in` reverse-lookups — out of scope per your call.
|
||||||
|
|
||||||
|
### Known gaps / risks (read before relying on it)
|
||||||
|
|
||||||
|
- It is **0.1.0**, written in one session. Treat its youth as the real risk.
|
||||||
|
- **No multi-document transactions** exposed (Couchbase has them in the SDK; the
|
||||||
|
ORM doesn't surface them yet).
|
||||||
|
- You **design GSIs deliberately** — a query with no supporting index does a full
|
||||||
|
primary scan. There is no migration framework for indexes yet.
|
||||||
|
- `embedded_in` is unimplemented; one flaky namespaced-ancestry test.
|
||||||
|
|
||||||
|
### One recommended pattern shift
|
||||||
|
|
||||||
|
v0 uses a single collection with a `ruby_class` discriminator (closest to
|
||||||
|
CouchDB; runs the whole suite as-is). For production, map each model to its own
|
||||||
|
Couchbase **Collection** — tighter GSIs, cleaner queries, idiomatic. A
|
||||||
|
`collection :name` per-model DSL is the planned opt-in.
|
||||||
|
|
||||||
|
### Integration sketch
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
# Gemfile
|
||||||
|
gem 'simply_based', git: 'https://github.com/gbentz2/simply_based.git'
|
||||||
|
|
||||||
|
# config/initializers/simply_based.rb
|
||||||
|
SimplyBased.database_url =
|
||||||
|
"couchbase://Administrator:password@localhost/app?scope=_default&collection=_default"
|
||||||
|
|
||||||
|
# config/environments/*.rb
|
||||||
|
config.cache_store = SimplyBased::Cache.new(namespace: "app")
|
||||||
|
```
|
||||||
|
```ruby
|
||||||
|
class User
|
||||||
|
include SimplyBased::Model
|
||||||
|
property :name
|
||||||
|
has_many :posts
|
||||||
|
validates :name, presence: true
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Should this ORM exist? — SimplyBased vs. Postgres + Redis
|
||||||
|
|
||||||
|
You raised the real question: another path is to drop CouchDB and refactor to a
|
||||||
|
standard Rails stack (Postgres + Redis). For an app with **many reads and writes
|
||||||
|
per second**, does a Couchbase-native ORM have real benefits? An honest take.
|
||||||
|
|
||||||
|
**Maturity first.** Postgres + ActiveRecord + Redis is the most battle-tested
|
||||||
|
stack in Rails. SimplyBased is 0.1.0 with known gaps. On risk and ecosystem
|
||||||
|
alone, the standard stack wins decisively. That sits at the top of the ledger.
|
||||||
|
|
||||||
|
| Dimension (sustained high throughput) | SimplyBased / Couchbase | Postgres + Redis |
|
||||||
|
|---|---|---|
|
||||||
|
| Raw write scale | memory-first, async persist, sub-ms KV, **shards horizontally** | single primary takes all writes; scales up, not out (shard/partition = heavy) |
|
||||||
|
| By-id reads | sub-ms from RAM | fast, network + planner hop; replicas for read scale |
|
||||||
|
| Complex/ad-hoc queries | N1QL, but **you design every GSI**; missing index = full scan | world-class planner, joins, ad-hoc SQL |
|
||||||
|
| Transactions / integrity | single-doc atomic + CAS only (multi-doc not exposed) | **ACID, FKs, constraints, multi-row txns** |
|
||||||
|
| Operational surface | **one datastore** — docs + cache + counters + locks, no Redis | two systems, two backups, PG↔Redis cache-invalidation drift |
|
||||||
|
| Cost / footprint | RAM-hungry, multi-service, doesn't scale down; some EE-licensed | both lean, cheap, scale down; fully open |
|
||||||
|
| Schema | schemaless documents, denormalized aggregates | migrations — but **`jsonb` gives documents *with* ACID** |
|
||||||
|
|
||||||
|
**Verdict.** Couchbase's write advantage is real but only **above a high ceiling**
|
||||||
|
— a tuned Postgres does tens of thousands of writes/sec; you only *need*
|
||||||
|
Couchbase's horizontal write scale-out when you'd otherwise be sharding Postgres
|
||||||
|
(the genuinely painful threshold). Below it, Postgres + Redis wins on
|
||||||
|
transactions, cost, ecosystem and hiring.
|
||||||
|
|
||||||
|
So it's not "which is faster" — it's **data shape and scale horizon**:
|
||||||
|
|
||||||
|
- **SimplyBased/Couchbase wins** when the domain is document-shaped and
|
||||||
|
denormalized, cross-entity transactions are rare, you truly expect to outgrow a
|
||||||
|
single Postgres primary, and the **one-datastore consolidation** (no Redis, no
|
||||||
|
cache-invalidation drift) is worth running a heavier engine.
|
||||||
|
- **Postgres + Redis wins** for relational/transactional domains (most business
|
||||||
|
apps), the mature default Rails path, cost-sensitivity, or when `jsonb` already
|
||||||
|
covers your document needs.
|
||||||
|
|
||||||
|
**Migration-cost asymmetry — and it favors SimplyBased for *this* project.** You
|
||||||
|
are already on CouchDB with simply_couch models:
|
||||||
|
|
||||||
|
- *SimplyBased path:* `include` swap, design GSIs, adopt the collection pattern,
|
||||||
|
copy data CouchDB→Couchbase. Document model + association API survive intact.
|
||||||
|
Low app-code churn.
|
||||||
|
- *Postgres path:* re-model documents into tables (or jsonb), rewrite every model
|
||||||
|
to ActiveRecord, rebuild associations as FKs, re-express views as SQL, rewire
|
||||||
|
jobs/cache. You land on the best-supported stack — but it's a large rewrite.
|
||||||
|
|
||||||
|
**The option neither of us named first, and often the strongest:**
|
||||||
|
**Postgres + `jsonb` + Redis.** Document flexibility *and* ACID, on the mature
|
||||||
|
Rails rails, with Redis still doing cache/counters/locks. For a transactional
|
||||||
|
high-throughput app this is the "boring tech wins" answer.
|
||||||
|
|
||||||
|
**Bottom line.** SimplyBased earns its place when the document model is the point,
|
||||||
|
write scale-out is a real future need, and consolidating onto one engine is worth
|
||||||
|
the heavier footprint and the youth of the library. For getting *this* project off
|
||||||
|
CouchDB with minimal model-code churn, it's the lowest-friction path and it's
|
||||||
|
genuinely working (403/403 applicable). But the benchmark gap only pays off past a
|
||||||
|
scale most apps never reach — so weigh the 0.1.0 maturity and the
|
||||||
|
multi-doc-transaction gap as the real decision factors, not the throughput
|
||||||
|
numbers.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repos & access
|
||||||
|
|
||||||
|
| Artifact | Location | State |
|
||||||
|
|---|---|---|
|
||||||
|
| Adapter refactor | PR https://github.com/bterkuile/simply_couch/pull/10 | open |
|
||||||
|
| Couchbase adapter gem | https://github.com/gbentz2/simply_couch-couchbase | public |
|
||||||
|
| SimplyBased ORM | https://github.com/gbentz2/simply_based | **private** |
|
||||||
|
|
||||||
|
To browse `simply_based`, the cleanest options are: Greg flips it public, adds
|
||||||
|
your `bterkuile` GitHub account as a collaborator, or mints a read-only
|
||||||
|
fine-grained deploy token scoped to that one repo. (A link with Greg's personal
|
||||||
|
token embedded works for `git clone` but hands over his whole account — not
|
||||||
|
recommended for anything durable.)
|
||||||
|
|
||||||
|
## Reproduce the verification
|
||||||
|
|
||||||
|
- Ruby 3.1+ (built on 4.0.5). `bundle install`.
|
||||||
|
- CouchDB: `docker run -d -p 5984:5984 -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=secret couchdb:3`
|
||||||
|
- Couchbase: `docker run -d -p 8091-8096:8091-8096 -p 11210:11210 couchbase:community`,
|
||||||
|
then init a single-node cluster + a **couchstore** bucket (Community edition
|
||||||
|
rejects the default magma backend) named `simply_based_test`.
|
||||||
|
- `COUCHBASE_URL="couchbase://Administrator:password@localhost/simply_based_test" bundle exec rspec`
|
||||||
Reference in new issue
Block a user