Add session report (architecture, coverage, ORM vs Postgres+Redis)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
GregandClaude Opus 4.8 committed
1 parent
45659c957e
commit
d2202bda03
1 file changed
+302
@@ -0,0 +1,302 @@
|
||||
# simply_couch → Couchbase: Session Report
|
||||
|
||||
**For:** Benjamin ter Kuile (bterkuile)
|
||||
**Prepared by:** Greg (via Claude Code / Opus 4.8)
|
||||
**Date:** 2026-06-20
|
||||
|
||||
This document summarizes a working session that started from one request —
|
||||
"compartmentalize simply_couch's data backend so CouchDB could be swapped for
|
||||
Couchbase" — and grew, through your input in the thread, into three concrete
|
||||
artifacts plus an architectural recommendation. Everything below was built and
|
||||
verified against a live database (CouchDB 3.5.2 and Couchbase Community 7.x in
|
||||
local Docker); nothing here is hypothetical.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
Three artifacts, in increasing ambition:
|
||||
|
||||
1. **PR #10 on `bterkuile/simply_couch`** — a behaviour-preserving refactor that
|
||||
turns the leaky single-driver seam into a real, documented adapter contract.
|
||||
Full suite green (453 examples). CouchDB still the default.
|
||||
2. **`simply_couch-couchbase`** — a Couchbase *adapter plugin* for the refactored
|
||||
simply_couch (kept the CouchDB-shaped API). SDK-based, N1QL view translation.
|
||||
3. **`simply_based`** — the *revolutionary* path you asked for: a standalone
|
||||
Couchbase-native ORM. `include SimplyBased::Model`, no CouchDB concepts left.
|
||||
Runs the simply_couch test suite: **403/403 applicable examples green**.
|
||||
|
||||
And a candid recommendation at the end: whether a Couchbase-native ORM is
|
||||
actually the right call versus a standard Postgres + Redis Rails stack.
|
||||
|
||||
---
|
||||
|
||||
## The core strategy point: Couchbase ≠ CouchDB
|
||||
|
||||
Despite the shared name and lineage (Damien Katz), the two are not API-compatible,
|
||||
and a literal `simply_couchbase_rest` drop-in for `couchrest` cannot exist:
|
||||
|
||||
| Concern | CouchDB (couchrest) | Couchbase |
|
||||
|---|---|---|
|
||||
| Data path | HTTP REST per-document | binary KV via SDK (REST is management-only) |
|
||||
| Query | JS map/reduce **design-doc views** | **N1QL / SQL++** + GSI (CB views deprecated in 7.0) |
|
||||
| Concurrency | MVCC `_rev`, conflict docs | CAS (uint64), no user-facing revisions |
|
||||
| Bulk | `_bulk_docs` | batched KV / subdoc ops |
|
||||
| Attachments | native blob attachments | none — store externally (S3/local) |
|
||||
| "CouchDB interface" | — | only the deprecated Views engine (port 8092 CAPI), or Sync Gateway's replication protocol — neither is a general server API for an ORM |
|
||||
|
||||
So Couchbase is reached the way it's meant to be: the SDK for KV, **N1QL** for
|
||||
queries. Not a REST shim. The pinch point is the view system — but simply_couch
|
||||
builds views from **structured Ruby spec objects**, not ad-hoc JS, so the finder
|
||||
and association queries translate deterministically to N1QL. That's what made the
|
||||
whole thing tractable.
|
||||
|
||||
---
|
||||
|
||||
## Artifact 1 — PR #10: the adapter-contract refactor
|
||||
|
||||
**Link:** https://github.com/bterkuile/simply_couch/pull/10 (open; you were
|
||||
notified on open).
|
||||
|
||||
**What it does (behaviour-preserving — zero functional change for couchrest users):**
|
||||
|
||||
- Adds `SimplyCouch::Adapter` — the backend-neutral contract the model layer
|
||||
depends on (`save_document`, `load_document`, `destroy_document`, `bulk_*`,
|
||||
`view`, `first`, `create_database!`, `drop_database!`), plus an exception
|
||||
contract (drivers translate their errors to `SimplyCouch::Conflict` /
|
||||
`NotFound`).
|
||||
- Moves the existing CouchDB code out of `Model::DatabaseInstance` into
|
||||
`SimplyCouch::Adapters::CouchRest` **verbatim** (a `DatabaseInstance` alias is
|
||||
kept). This is a move, not a rewrite.
|
||||
- **Lazy driver loading:** `require 'simply_couch'` now loads with no driver
|
||||
present; couchrest loads only when its adapter is first used.
|
||||
- Adapter registry: `SimplyCouch.register_adapter(name, class, require_path:)`
|
||||
+ `SimplyCouch.adapter = :name`.
|
||||
- Stops the model/association/view layers reaching through
|
||||
`database.couchrest_database` (5 sites → the neutral `database.bulk_load`).
|
||||
- `SimplyCouch::DocumentCodec` — rebuilds a model from a plain hash via the
|
||||
`ruby_class` tag, so non-couchrest backends can hydrate documents.
|
||||
- Exposes the view-spec `klass`/`options` as the backend-neutral query IR.
|
||||
- Hardens the load guard so a plugin gem can open `module SimplyCouch` for its
|
||||
own namespace before the library loads.
|
||||
- Adds shared `'a simply_couch adapter'` RSpec examples (reusable parity proof).
|
||||
- Version 0.1.2 → 0.2.0; README/CHANGELOG updated.
|
||||
|
||||
**Verification:** 453 examples, 0 failures vs a live CouchDB 3.5.2. No model-facing
|
||||
signature changed. (Two pre-existing flaky DB-reset races predate the PR.)
|
||||
|
||||
---
|
||||
|
||||
## Artifact 2 — `simply_couch-couchbase` (adapter plugin)
|
||||
|
||||
A separate gem that plugs the **refactored** simply_couch into Couchbase while
|
||||
keeping the CouchDB-shaped API. `SimplyCouch::Adapters::Couchbase` on the official
|
||||
`couchbase` SDK: KV for documents, CAS↔`_rev`, and an `N1qlTranslator` that
|
||||
renders the structured `ModelViewSpec` / `DeletedModelViewSpec` (all_documents /
|
||||
find_by / count) to N1QL. Custom/raw JS association views raise
|
||||
`SimplyCouch::UnsupportedView` (documented boundary).
|
||||
|
||||
**Verification:** 20 examples green = 11 N1QL-translator unit tests (no server) +
|
||||
9 live-Couchbase contract examples (the *same* shared examples couchrest passes).
|
||||
|
||||
This is the lower-ambition path: it proves the adapter seam works and gives a
|
||||
working Couchbase backend, but associations were left as an explicit boundary.
|
||||
|
||||
---
|
||||
|
||||
## Artifact 3 — `simply_based` (the Couchbase-native ORM)
|
||||
|
||||
The path you chose in the thread: not an adapter, a **new gem** that *is* the
|
||||
ORM, Couchbase-native throughout.
|
||||
|
||||
**The one-line swap:**
|
||||
```ruby
|
||||
- include SimplyCouch::Model
|
||||
+ include SimplyBased::Model
|
||||
```
|
||||
|
||||
**Why so little changes:** simply_couch's model layer (properties, type-casting,
|
||||
validations, callbacks, dirty tracking, the association DSL, soft-delete,
|
||||
pagination) is already storage-neutral. SimplyBased keeps all of it and replaces
|
||||
exactly one layer — `SimplyBased::Store` — which speaks Couchbase directly. Every
|
||||
association lookup funnels through a handful of Store methods that emit N1QL.
|
||||
|
||||
**How the CouchDB views became N1QL:**
|
||||
|
||||
- `belongs_to` / `has_many` / `has_one` → scalar foreign-key equality:
|
||||
`WHERE ruby_class = $rc AND user_id = $id`, ordered by `created_at`, GSI auto-
|
||||
created on first use.
|
||||
- `has_and_belongs_to_many` → on the storing side, a **KV multi-get** of the id
|
||||
array (no query at all); on the other side, the two hand-written map/reduce
|
||||
functions become one query:
|
||||
```sql
|
||||
SELECT c.*, META(c).id, META(c).cas
|
||||
FROM `bucket`.`_default`.`_default` AS c
|
||||
WHERE c.ruby_class = $rc
|
||||
AND ANY x IN c.`network_ids` SATISFIES x = $v END
|
||||
```
|
||||
backed by an array index.
|
||||
- **Ancestry / trees** → you were right that it's a nested-set/materialized-path
|
||||
intention. `path_ids` is the full path root→self, so no recursive build:
|
||||
- children = `path_ids[ARRAY_LENGTH-2] = $id`
|
||||
- subtree = `$id IN path_ids` (minus self)
|
||||
- roots = `ARRAY_LENGTH(path_ids) = 1`
|
||||
- `_rev` is mapped onto **CAS**: a record's `_rev` is the stringified CAS of its
|
||||
last write; updates use it for optimistic locking (stale write →
|
||||
`SimplyBased::Conflict`).
|
||||
- Soft-delete-aware `find_by`/`find_all_by`: the `[key, deleted_at] = [val, nil]`
|
||||
convention compares element-wise so a `nil` component matches `MISSING`/`NULL`
|
||||
(non-deleted docs carry no `deleted_at`).
|
||||
- Timestamps are microsecond-precision so same-second creations paginate in
|
||||
creation order deterministically.
|
||||
|
||||
**Replaces Redis too** (same cluster, no second datastore) — 5/5 green:
|
||||
|
||||
- `SimplyBased::Counter` — Couchbase binary increment/decrement (true atomic
|
||||
counters; a drop-in for Redis INCR/DECR).
|
||||
- `SimplyBased::Cache` — an `ActiveSupport::Cache::Store` over KV + native TTL;
|
||||
usable as `Rails.cache` and therefore the ActiveJob / fragment / low-level cache.
|
||||
- `SimplyBased::Lock` — distributed lock via insert-if-absent + TTL auto-expiry +
|
||||
CAS-checked release (only the holder can unlock).
|
||||
|
||||
### Test coverage
|
||||
|
||||
SimplyBased runs the **actual simply_couch test suite** against a live Couchbase.
|
||||
|
||||
- **403 / 403 applicable examples green on a clean run.** One ancestry reparent
|
||||
edge in the namespaced (`by_property`) cascade is intermittently flaky (~50%) —
|
||||
it does several dependent saves and occasionally races; tracked, not yet nailed.
|
||||
- **Green:** CRUD; finders (`find`, `all`, `first`, `find_by_*`, `count_by_*`);
|
||||
`belongs_to` / `has_many` / `has_one` / `has_and_belongs_to_many`; validations;
|
||||
callbacks; dirty tracking; soft-delete (incl. soft-delete-aware finders);
|
||||
pagination; bulk; eager-loading; ancestry/trees; the Redis primitives.
|
||||
- **Excluded by design:**
|
||||
- `:couchdb_only` — native inline attachments, JS-view token-safety,
|
||||
design-document maintenance, raw custom JS views. No Couchbase equivalent
|
||||
(use `has_s3_attached` / `has_local_attached`; N1QL uses bound parameters).
|
||||
- `embedded_in` reverse-lookups — out of scope per your call.
|
||||
|
||||
### Known gaps / risks (read before relying on it)
|
||||
|
||||
- It is **0.1.0**, written in one session. Treat its youth as the real risk.
|
||||
- **No multi-document transactions** exposed (Couchbase has them in the SDK; the
|
||||
ORM doesn't surface them yet).
|
||||
- You **design GSIs deliberately** — a query with no supporting index does a full
|
||||
primary scan. There is no migration framework for indexes yet.
|
||||
- `embedded_in` is unimplemented; one flaky namespaced-ancestry test.
|
||||
|
||||
### One recommended pattern shift
|
||||
|
||||
v0 uses a single collection with a `ruby_class` discriminator (closest to
|
||||
CouchDB; runs the whole suite as-is). For production, map each model to its own
|
||||
Couchbase **Collection** — tighter GSIs, cleaner queries, idiomatic. A
|
||||
`collection :name` per-model DSL is the planned opt-in.
|
||||
|
||||
### Integration sketch
|
||||
|
||||
```ruby
|
||||
# Gemfile
|
||||
gem 'simply_based', git: 'https://github.com/gbentz2/simply_based.git'
|
||||
|
||||
# config/initializers/simply_based.rb
|
||||
SimplyBased.database_url =
|
||||
"couchbase://Administrator:password@localhost/app?scope=_default&collection=_default"
|
||||
|
||||
# config/environments/*.rb
|
||||
config.cache_store = SimplyBased::Cache.new(namespace: "app")
|
||||
```
|
||||
```ruby
|
||||
class User
|
||||
include SimplyBased::Model
|
||||
property :name
|
||||
has_many :posts
|
||||
validates :name, presence: true
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Should this ORM exist? — SimplyBased vs. Postgres + Redis
|
||||
|
||||
You raised the real question: another path is to drop CouchDB and refactor to a
|
||||
standard Rails stack (Postgres + Redis). For an app with **many reads and writes
|
||||
per second**, does a Couchbase-native ORM have real benefits? An honest take.
|
||||
|
||||
**Maturity first.** Postgres + ActiveRecord + Redis is the most battle-tested
|
||||
stack in Rails. SimplyBased is 0.1.0 with known gaps. On risk and ecosystem
|
||||
alone, the standard stack wins decisively. That sits at the top of the ledger.
|
||||
|
||||
| Dimension (sustained high throughput) | SimplyBased / Couchbase | Postgres + Redis |
|
||||
|---|---|---|
|
||||
| Raw write scale | memory-first, async persist, sub-ms KV, **shards horizontally** | single primary takes all writes; scales up, not out (shard/partition = heavy) |
|
||||
| By-id reads | sub-ms from RAM | fast, network + planner hop; replicas for read scale |
|
||||
| Complex/ad-hoc queries | N1QL, but **you design every GSI**; missing index = full scan | world-class planner, joins, ad-hoc SQL |
|
||||
| Transactions / integrity | single-doc atomic + CAS only (multi-doc not exposed) | **ACID, FKs, constraints, multi-row txns** |
|
||||
| Operational surface | **one datastore** — docs + cache + counters + locks, no Redis | two systems, two backups, PG↔Redis cache-invalidation drift |
|
||||
| Cost / footprint | RAM-hungry, multi-service, doesn't scale down; some EE-licensed | both lean, cheap, scale down; fully open |
|
||||
| Schema | schemaless documents, denormalized aggregates | migrations — but **`jsonb` gives documents *with* ACID** |
|
||||
|
||||
**Verdict.** Couchbase's write advantage is real but only **above a high ceiling**
|
||||
— a tuned Postgres does tens of thousands of writes/sec; you only *need*
|
||||
Couchbase's horizontal write scale-out when you'd otherwise be sharding Postgres
|
||||
(the genuinely painful threshold). Below it, Postgres + Redis wins on
|
||||
transactions, cost, ecosystem and hiring.
|
||||
|
||||
So it's not "which is faster" — it's **data shape and scale horizon**:
|
||||
|
||||
- **SimplyBased/Couchbase wins** when the domain is document-shaped and
|
||||
denormalized, cross-entity transactions are rare, you truly expect to outgrow a
|
||||
single Postgres primary, and the **one-datastore consolidation** (no Redis, no
|
||||
cache-invalidation drift) is worth running a heavier engine.
|
||||
- **Postgres + Redis wins** for relational/transactional domains (most business
|
||||
apps), the mature default Rails path, cost-sensitivity, or when `jsonb` already
|
||||
covers your document needs.
|
||||
|
||||
**Migration-cost asymmetry — and it favors SimplyBased for *this* project.** You
|
||||
are already on CouchDB with simply_couch models:
|
||||
|
||||
- *SimplyBased path:* `include` swap, design GSIs, adopt the collection pattern,
|
||||
copy data CouchDB→Couchbase. Document model + association API survive intact.
|
||||
Low app-code churn.
|
||||
- *Postgres path:* re-model documents into tables (or jsonb), rewrite every model
|
||||
to ActiveRecord, rebuild associations as FKs, re-express views as SQL, rewire
|
||||
jobs/cache. You land on the best-supported stack — but it's a large rewrite.
|
||||
|
||||
**The option neither of us named first, and often the strongest:**
|
||||
**Postgres + `jsonb` + Redis.** Document flexibility *and* ACID, on the mature
|
||||
Rails rails, with Redis still doing cache/counters/locks. For a transactional
|
||||
high-throughput app this is the "boring tech wins" answer.
|
||||
|
||||
**Bottom line.** SimplyBased earns its place when the document model is the point,
|
||||
write scale-out is a real future need, and consolidating onto one engine is worth
|
||||
the heavier footprint and the youth of the library. For getting *this* project off
|
||||
CouchDB with minimal model-code churn, it's the lowest-friction path and it's
|
||||
genuinely working (403/403 applicable). But the benchmark gap only pays off past a
|
||||
scale most apps never reach — so weigh the 0.1.0 maturity and the
|
||||
multi-doc-transaction gap as the real decision factors, not the throughput
|
||||
numbers.
|
||||
|
||||
---
|
||||
|
||||
## Repos & access
|
||||
|
||||
| Artifact | Location | State |
|
||||
|---|---|---|
|
||||
| Adapter refactor | PR https://github.com/bterkuile/simply_couch/pull/10 | open |
|
||||
| Couchbase adapter gem | https://github.com/gbentz2/simply_couch-couchbase | public |
|
||||
| SimplyBased ORM | https://github.com/gbentz2/simply_based | **private** |
|
||||
|
||||
To browse `simply_based`, the cleanest options are: Greg flips it public, adds
|
||||
your `bterkuile` GitHub account as a collaborator, or mints a read-only
|
||||
fine-grained deploy token scoped to that one repo. (A link with Greg's personal
|
||||
token embedded works for `git clone` but hands over his whole account — not
|
||||
recommended for anything durable.)
|
||||
|
||||
## Reproduce the verification
|
||||
|
||||
- Ruby 3.1+ (built on 4.0.5). `bundle install`.
|
||||
- CouchDB: `docker run -d -p 5984:5984 -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=secret couchdb:3`
|
||||
- Couchbase: `docker run -d -p 8091-8096:8091-8096 -p 11210:11210 couchbase:community`,
|
||||
then init a single-node cluster + a **couchstore** bucket (Community edition
|
||||
rejects the default magma backend) named `simply_based_test`.
|
||||
- `COUCHBASE_URL="couchbase://Administrator:password@localhost/simply_based_test" bundle exec rspec`
|
||||
Reference in new issue
Block a user