diff --git a/SESSION_REPORT.md b/SESSION_REPORT.md new file mode 100644 index 0000000..8d2779d --- /dev/null +++ b/SESSION_REPORT.md @@ -0,0 +1,302 @@ +# simply_couch → Couchbase: Session Report + +**For:** Benjamin ter Kuile (bterkuile) +**Prepared by:** Greg (via Claude Code / Opus 4.8) +**Date:** 2026-06-20 + +This document summarizes a working session that started from one request — +"compartmentalize simply_couch's data backend so CouchDB could be swapped for +Couchbase" — and grew, through your input in the thread, into three concrete +artifacts plus an architectural recommendation. Everything below was built and +verified against a live database (CouchDB 3.5.2 and Couchbase Community 7.x in +local Docker); nothing here is hypothetical. + +--- + +## TL;DR + +Three artifacts, in increasing ambition: + +1. **PR #10 on `bterkuile/simply_couch`** — a behaviour-preserving refactor that + turns the leaky single-driver seam into a real, documented adapter contract. + Full suite green (453 examples). CouchDB still the default. +2. **`simply_couch-couchbase`** — a Couchbase *adapter plugin* for the refactored + simply_couch (kept the CouchDB-shaped API). SDK-based, N1QL view translation. +3. **`simply_based`** — the *revolutionary* path you asked for: a standalone + Couchbase-native ORM. `include SimplyBased::Model`, no CouchDB concepts left. + Runs the simply_couch test suite: **403/403 applicable examples green**. + +And a candid recommendation at the end: whether a Couchbase-native ORM is +actually the right call versus a standard Postgres + Redis Rails stack. + +--- + +## The core strategy point: Couchbase ≠ CouchDB + +Despite the shared name and lineage (Damien Katz), the two are not API-compatible, +and a literal `simply_couchbase_rest` drop-in for `couchrest` cannot exist: + +| Concern | CouchDB (couchrest) | Couchbase | +|---|---|---| +| Data path | HTTP REST per-document | binary KV via SDK (REST is management-only) | +| Query | JS map/reduce **design-doc views** | **N1QL / SQL++** + GSI (CB views deprecated in 7.0) | +| Concurrency | MVCC `_rev`, conflict docs | CAS (uint64), no user-facing revisions | +| Bulk | `_bulk_docs` | batched KV / subdoc ops | +| Attachments | native blob attachments | none — store externally (S3/local) | +| "CouchDB interface" | — | only the deprecated Views engine (port 8092 CAPI), or Sync Gateway's replication protocol — neither is a general server API for an ORM | + +So Couchbase is reached the way it's meant to be: the SDK for KV, **N1QL** for +queries. Not a REST shim. The pinch point is the view system — but simply_couch +builds views from **structured Ruby spec objects**, not ad-hoc JS, so the finder +and association queries translate deterministically to N1QL. That's what made the +whole thing tractable. + +--- + +## Artifact 1 — PR #10: the adapter-contract refactor + +**Link:** https://github.com/bterkuile/simply_couch/pull/10 (open; you were +notified on open). + +**What it does (behaviour-preserving — zero functional change for couchrest users):** + +- Adds `SimplyCouch::Adapter` — the backend-neutral contract the model layer + depends on (`save_document`, `load_document`, `destroy_document`, `bulk_*`, + `view`, `first`, `create_database!`, `drop_database!`), plus an exception + contract (drivers translate their errors to `SimplyCouch::Conflict` / + `NotFound`). +- Moves the existing CouchDB code out of `Model::DatabaseInstance` into + `SimplyCouch::Adapters::CouchRest` **verbatim** (a `DatabaseInstance` alias is + kept). This is a move, not a rewrite. +- **Lazy driver loading:** `require 'simply_couch'` now loads with no driver + present; couchrest loads only when its adapter is first used. +- Adapter registry: `SimplyCouch.register_adapter(name, class, require_path:)` + + `SimplyCouch.adapter = :name`. +- Stops the model/association/view layers reaching through + `database.couchrest_database` (5 sites → the neutral `database.bulk_load`). +- `SimplyCouch::DocumentCodec` — rebuilds a model from a plain hash via the + `ruby_class` tag, so non-couchrest backends can hydrate documents. +- Exposes the view-spec `klass`/`options` as the backend-neutral query IR. +- Hardens the load guard so a plugin gem can open `module SimplyCouch` for its + own namespace before the library loads. +- Adds shared `'a simply_couch adapter'` RSpec examples (reusable parity proof). +- Version 0.1.2 → 0.2.0; README/CHANGELOG updated. + +**Verification:** 453 examples, 0 failures vs a live CouchDB 3.5.2. No model-facing +signature changed. (Two pre-existing flaky DB-reset races predate the PR.) + +--- + +## Artifact 2 — `simply_couch-couchbase` (adapter plugin) + +A separate gem that plugs the **refactored** simply_couch into Couchbase while +keeping the CouchDB-shaped API. `SimplyCouch::Adapters::Couchbase` on the official +`couchbase` SDK: KV for documents, CAS↔`_rev`, and an `N1qlTranslator` that +renders the structured `ModelViewSpec` / `DeletedModelViewSpec` (all_documents / +find_by / count) to N1QL. Custom/raw JS association views raise +`SimplyCouch::UnsupportedView` (documented boundary). + +**Verification:** 20 examples green = 11 N1QL-translator unit tests (no server) + +9 live-Couchbase contract examples (the *same* shared examples couchrest passes). + +This is the lower-ambition path: it proves the adapter seam works and gives a +working Couchbase backend, but associations were left as an explicit boundary. + +--- + +## Artifact 3 — `simply_based` (the Couchbase-native ORM) + +The path you chose in the thread: not an adapter, a **new gem** that *is* the +ORM, Couchbase-native throughout. + +**The one-line swap:** +```ruby +- include SimplyCouch::Model ++ include SimplyBased::Model +``` + +**Why so little changes:** simply_couch's model layer (properties, type-casting, +validations, callbacks, dirty tracking, the association DSL, soft-delete, +pagination) is already storage-neutral. SimplyBased keeps all of it and replaces +exactly one layer — `SimplyBased::Store` — which speaks Couchbase directly. Every +association lookup funnels through a handful of Store methods that emit N1QL. + +**How the CouchDB views became N1QL:** + +- `belongs_to` / `has_many` / `has_one` → scalar foreign-key equality: + `WHERE ruby_class = $rc AND user_id = $id`, ordered by `created_at`, GSI auto- + created on first use. +- `has_and_belongs_to_many` → on the storing side, a **KV multi-get** of the id + array (no query at all); on the other side, the two hand-written map/reduce + functions become one query: + ```sql + SELECT c.*, META(c).id, META(c).cas + FROM `bucket`.`_default`.`_default` AS c + WHERE c.ruby_class = $rc + AND ANY x IN c.`network_ids` SATISFIES x = $v END + ``` + backed by an array index. +- **Ancestry / trees** → you were right that it's a nested-set/materialized-path + intention. `path_ids` is the full path root→self, so no recursive build: + - children = `path_ids[ARRAY_LENGTH-2] = $id` + - subtree = `$id IN path_ids` (minus self) + - roots = `ARRAY_LENGTH(path_ids) = 1` +- `_rev` is mapped onto **CAS**: a record's `_rev` is the stringified CAS of its + last write; updates use it for optimistic locking (stale write → + `SimplyBased::Conflict`). +- Soft-delete-aware `find_by`/`find_all_by`: the `[key, deleted_at] = [val, nil]` + convention compares element-wise so a `nil` component matches `MISSING`/`NULL` + (non-deleted docs carry no `deleted_at`). +- Timestamps are microsecond-precision so same-second creations paginate in + creation order deterministically. + +**Replaces Redis too** (same cluster, no second datastore) — 5/5 green: + +- `SimplyBased::Counter` — Couchbase binary increment/decrement (true atomic + counters; a drop-in for Redis INCR/DECR). +- `SimplyBased::Cache` — an `ActiveSupport::Cache::Store` over KV + native TTL; + usable as `Rails.cache` and therefore the ActiveJob / fragment / low-level cache. +- `SimplyBased::Lock` — distributed lock via insert-if-absent + TTL auto-expiry + + CAS-checked release (only the holder can unlock). + +### Test coverage + +SimplyBased runs the **actual simply_couch test suite** against a live Couchbase. + +- **403 / 403 applicable examples green on a clean run.** One ancestry reparent + edge in the namespaced (`by_property`) cascade is intermittently flaky (~50%) — + it does several dependent saves and occasionally races; tracked, not yet nailed. +- **Green:** CRUD; finders (`find`, `all`, `first`, `find_by_*`, `count_by_*`); + `belongs_to` / `has_many` / `has_one` / `has_and_belongs_to_many`; validations; + callbacks; dirty tracking; soft-delete (incl. soft-delete-aware finders); + pagination; bulk; eager-loading; ancestry/trees; the Redis primitives. +- **Excluded by design:** + - `:couchdb_only` — native inline attachments, JS-view token-safety, + design-document maintenance, raw custom JS views. No Couchbase equivalent + (use `has_s3_attached` / `has_local_attached`; N1QL uses bound parameters). + - `embedded_in` reverse-lookups — out of scope per your call. + +### Known gaps / risks (read before relying on it) + +- It is **0.1.0**, written in one session. Treat its youth as the real risk. +- **No multi-document transactions** exposed (Couchbase has them in the SDK; the + ORM doesn't surface them yet). +- You **design GSIs deliberately** — a query with no supporting index does a full + primary scan. There is no migration framework for indexes yet. +- `embedded_in` is unimplemented; one flaky namespaced-ancestry test. + +### One recommended pattern shift + +v0 uses a single collection with a `ruby_class` discriminator (closest to +CouchDB; runs the whole suite as-is). For production, map each model to its own +Couchbase **Collection** — tighter GSIs, cleaner queries, idiomatic. A +`collection :name` per-model DSL is the planned opt-in. + +### Integration sketch + +```ruby +# Gemfile +gem 'simply_based', git: 'https://github.com/gbentz2/simply_based.git' + +# config/initializers/simply_based.rb +SimplyBased.database_url = + "couchbase://Administrator:password@localhost/app?scope=_default&collection=_default" + +# config/environments/*.rb +config.cache_store = SimplyBased::Cache.new(namespace: "app") +``` +```ruby +class User + include SimplyBased::Model + property :name + has_many :posts + validates :name, presence: true +end +``` + +--- + +## Should this ORM exist? — SimplyBased vs. Postgres + Redis + +You raised the real question: another path is to drop CouchDB and refactor to a +standard Rails stack (Postgres + Redis). For an app with **many reads and writes +per second**, does a Couchbase-native ORM have real benefits? An honest take. + +**Maturity first.** Postgres + ActiveRecord + Redis is the most battle-tested +stack in Rails. SimplyBased is 0.1.0 with known gaps. On risk and ecosystem +alone, the standard stack wins decisively. That sits at the top of the ledger. + +| Dimension (sustained high throughput) | SimplyBased / Couchbase | Postgres + Redis | +|---|---|---| +| Raw write scale | memory-first, async persist, sub-ms KV, **shards horizontally** | single primary takes all writes; scales up, not out (shard/partition = heavy) | +| By-id reads | sub-ms from RAM | fast, network + planner hop; replicas for read scale | +| Complex/ad-hoc queries | N1QL, but **you design every GSI**; missing index = full scan | world-class planner, joins, ad-hoc SQL | +| Transactions / integrity | single-doc atomic + CAS only (multi-doc not exposed) | **ACID, FKs, constraints, multi-row txns** | +| Operational surface | **one datastore** — docs + cache + counters + locks, no Redis | two systems, two backups, PG↔Redis cache-invalidation drift | +| Cost / footprint | RAM-hungry, multi-service, doesn't scale down; some EE-licensed | both lean, cheap, scale down; fully open | +| Schema | schemaless documents, denormalized aggregates | migrations — but **`jsonb` gives documents *with* ACID** | + +**Verdict.** Couchbase's write advantage is real but only **above a high ceiling** +— a tuned Postgres does tens of thousands of writes/sec; you only *need* +Couchbase's horizontal write scale-out when you'd otherwise be sharding Postgres +(the genuinely painful threshold). Below it, Postgres + Redis wins on +transactions, cost, ecosystem and hiring. + +So it's not "which is faster" — it's **data shape and scale horizon**: + +- **SimplyBased/Couchbase wins** when the domain is document-shaped and + denormalized, cross-entity transactions are rare, you truly expect to outgrow a + single Postgres primary, and the **one-datastore consolidation** (no Redis, no + cache-invalidation drift) is worth running a heavier engine. +- **Postgres + Redis wins** for relational/transactional domains (most business + apps), the mature default Rails path, cost-sensitivity, or when `jsonb` already + covers your document needs. + +**Migration-cost asymmetry — and it favors SimplyBased for *this* project.** You +are already on CouchDB with simply_couch models: + +- *SimplyBased path:* `include` swap, design GSIs, adopt the collection pattern, + copy data CouchDB→Couchbase. Document model + association API survive intact. + Low app-code churn. +- *Postgres path:* re-model documents into tables (or jsonb), rewrite every model + to ActiveRecord, rebuild associations as FKs, re-express views as SQL, rewire + jobs/cache. You land on the best-supported stack — but it's a large rewrite. + +**The option neither of us named first, and often the strongest:** +**Postgres + `jsonb` + Redis.** Document flexibility *and* ACID, on the mature +Rails rails, with Redis still doing cache/counters/locks. For a transactional +high-throughput app this is the "boring tech wins" answer. + +**Bottom line.** SimplyBased earns its place when the document model is the point, +write scale-out is a real future need, and consolidating onto one engine is worth +the heavier footprint and the youth of the library. For getting *this* project off +CouchDB with minimal model-code churn, it's the lowest-friction path and it's +genuinely working (403/403 applicable). But the benchmark gap only pays off past a +scale most apps never reach — so weigh the 0.1.0 maturity and the +multi-doc-transaction gap as the real decision factors, not the throughput +numbers. + +--- + +## Repos & access + +| Artifact | Location | State | +|---|---|---| +| Adapter refactor | PR https://github.com/bterkuile/simply_couch/pull/10 | open | +| Couchbase adapter gem | https://github.com/gbentz2/simply_couch-couchbase | public | +| SimplyBased ORM | https://github.com/gbentz2/simply_based | **private** | + +To browse `simply_based`, the cleanest options are: Greg flips it public, adds +your `bterkuile` GitHub account as a collaborator, or mints a read-only +fine-grained deploy token scoped to that one repo. (A link with Greg's personal +token embedded works for `git clone` but hands over his whole account — not +recommended for anything durable.) + +## Reproduce the verification + +- Ruby 3.1+ (built on 4.0.5). `bundle install`. +- CouchDB: `docker run -d -p 5984:5984 -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=secret couchdb:3` +- Couchbase: `docker run -d -p 8091-8096:8091-8096 -p 11210:11210 couchbase:community`, + then init a single-node cluster + a **couchstore** bucket (Community edition + rejects the default magma backend) named `simply_based_test`. +- `COUCHBASE_URL="couchbase://Administrator:password@localhost/simply_based_test" bundle exec rspec`