From f084e89991f2e1e941c0b8abe2899d6b7506238f Mon Sep 17 00:00:00 2001 From: goodboy Date: Wed, 10 Jun 2026 13:38:50 -0400 Subject: [PATCH] Add `datad`-split design plan + provenance docs Commit the AI-authored design doc driving this branch's 7-commit `brokerd` -> (`datad` + `brokerd`) split; all prior commits' `Prompt-IO:` entries reference its stages so this makes those refs resolvable in-repo for PR review. Also, - add the doc's own prompt-io entry pair (scope: docs) incl. a 4-item implementation-deviation log vs. the plan as written. (this patch was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code Co-Authored-By: Claude Fable 5 Prompt-IO: ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md --- ai/claude-code/plans/datad_service.md | 143 ++++++++++++++++++ .../20260610T173309Z_f15f8178_prompt_io.md | 66 ++++++++ ...20260610T173309Z_f15f8178_prompt_io.raw.md | 51 +++++++ 3 files changed, 260 insertions(+) create mode 100644 ai/claude-code/plans/datad_service.md create mode 100644 ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md create mode 100644 ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.raw.md diff --git a/ai/claude-code/plans/datad_service.md b/ai/claude-code/plans/datad_service.md new file mode 100644 index 00000000..76ab750b --- /dev/null +++ b/ai/claude-code/plans/datad_service.md @@ -0,0 +1,143 @@ +# Split `brokerd.` into trading-only `brokerd` + new `datad.` + +## Context + +Today a single `brokerd.` actor hosts BOTH concerns: + +- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers//feed.py`/`symbols.py`, +- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers//broker.py`, driven by `emsd`. + +The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split. + +**User-decided constraints:** + +- `datad.` is a **sibling** of `brokerd.` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`). +- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working. +- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes. + +## Target topology + +``` +pikerd +├── datad.ib feed bus, shm writers, tsp history, symbol search +├── brokerd.ib open_trade_dialog only (EMS-spawned, lazy) +├── emsd +│ └── paperboi.ib (paper mode; opens its feed via datad) +└── samplerd +``` + +## Key verified facts (load-bearing) + +- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor. +- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree. +- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param. +- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS. +- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns). +- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`. +- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split. +- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves. +- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad. + +## Module decision + +New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication). + +--- + +## Stage 0 — prep: backend module grouping + ep introspection (no topology change) + +1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks. +2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged: + - `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']` + - `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']` + - `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']` + - `ib`: adjust existing groups so each includes `'api'` + - flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior). +3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`). + +**Gate**: full pytest green, zero behavior change. + +## Stage 1 — introduce `datad` daemon machinery (additive) + +New `piker/data/_daemon.py`: + +- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`. +- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`. +- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`. +- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`. +- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`). + +Supporting edits: + +- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`). +- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`). +- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.) + +**Gate**: suite green + new spawn test; nothing routes through datad yet. + +## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`) + +Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal). + +`piker/clearing/_ems.py`: + +1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it. +2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`). +3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`. + +Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry. + +**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke. + +## Stage 3 — feed layer hard cutover to datad + +1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep. +2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.) +3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`: + - `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`) + - `piker/brokers/core.py:33,175` (`symbol_search`) + - `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`) + - `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'` +4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`. + +**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane. + +## Stage 4 — caps-sec slimming, validation, ib client-id + +1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all. +2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously. +3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys. +4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring. + +**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`. + +--- + +## Risk register + +| Risk | Sev | Mitigation | +|---|---|---| +| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns | +| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test | +| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast | +| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only | +| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it | +| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly | + +## Verification (per stage gates above, plus end-to-end) + +- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed). +- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger .paper`. +- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4). + +## Critical files + +- **new** `piker/data/_daemon.py` — all datad machinery +- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn +- `piker/data/feed.py` — spawn cutover + actor-name assert +- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`) +- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports +- `piker/data/validate.py` — `get_eps()` helper +- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods` +- `piker/accounting/cli.py` — keyword-form dialog call +- `tests/test_services.py` — inverted + new assertions diff --git a/ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md b/ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md new file mode 100644 index 00000000..db77be2d --- /dev/null +++ b/ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md @@ -0,0 +1,66 @@ +--- +model: claude-fable-5[1m] +service: claude +session: 32d15f9a-b2d3-4c26-bdc9-190219141a25 +timestamp: 2026-06-10T17:33:09Z +git_ref: datad_service +diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md +scope: docs +substantive: true +raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md +--- + +## Prompt + +The session-initiating instruction, verbatim (this doc +IS its "mega detailed plan" deliverable): + +> ok i want you to become the distributed runtime and +> concurrency expert for this project - namely acquire +> a deep understanding of tractor and how it's used. +> then i want you to atttempt to factor our current +> brokerd service daemon into 2 daemons: +> +> - a brokerd which only hosts live/paper trading +> endoint tasks as defined namely within all +> piker/brokers//broker.py mods +> - a new `datad` subdaemon which in a separate +> subactor serves all the data feed service tasks +> namely delivered by endpoints in +> piker/brokers//feed.py (or similar for +> less mature backends) and as more rigorously +> defined by the validation machinery in +> piker.data.validate. +> +> give me a mega detailed plan on how to approach +> this, and a staged approach for the implementation. + +Plan-shaping user decisions (in-session Q&A): sibling +topology under `pikerd`; hard cutover staged by layer; +trading-only EMS-lazy-spawned `brokerd`. The human +staged the doc copy into `ai/claude-code/plans/` and +requested this commit after the 7 implementation +commits landed. + +## Response summary + +The staged design doc for the `brokerd` -> (`datad` + +`brokerd`) split: context, target supervision topology, +load-bearing verified facts, per-stage file-level +changes with gates, risk register and an end-to-end +verification matrix. Produced via 3 parallel explore +agents + 1 architect agent + human Q&A before any code +was written; all 7 implementation commits in this +branch reference its stages in their provenance +entries. + +## Files changed + +- `ai/claude-code/plans/datad_service.md` — the design + plan doc (human-staged copy of the AI-authored plan) + +## Human edits + +None to the doc content — committed as generated. NB: +the implementation deviated from the plan-as-written in +4 places, see the raw file's deviation log. diff --git a/ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.raw.md b/ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.raw.md new file mode 100644 index 00000000..3c86de37 --- /dev/null +++ b/ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.raw.md @@ -0,0 +1,51 @@ +--- +model: claude-fable-5[1m] +service: claude +timestamp: 2026-06-10T17:33:09Z +git_ref: datad_service +diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md +--- + +NOTE: diff-ref mode entry; the AI output here is the +committed doc itself (a design plan, not code) so no +verbatim copy is duplicated: + +> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md` + +## Implementation deviation log + +Where the landed 7-commit series differs from the plan +as written (recorded for reviewer transparency; the +plan doc is committed unmodified): + +1. Commit ordering: the `pytest` config-dir isolation + fix lands BEFORE the `tractor`-API drift port. The + plan's stage gates assumed a runnable+isolated test + baseline; per-commit gating exposed that without + isolation the paper-EMS test reads the user's real + (polluted) `account.kraken.paper.toml` and reddens. +2. The plan's "stage 0: full pytest green" gate + required first repairing pre-existing branch + breakage vs `tractor` git `main` (boot + `AttributeError`, stale discovery/exc/position + APIs) — that repair became its own commit + ("Port service+tests to latest `tractor` APIs") + rather than plan-stage work. +3. Stage-4 fail-fast placement: the plan said + `broker_init()` raises on brokerd-ep-less backends; + implementation moved the raise to `spawn_brokerd()` + since `piker ledger` calls `broker_init()` directly + even for paper accounts on datad-only backends. +4. `brokers/_daemon.py` change grouping: the + import-cleanup hunks (`exceptiongroup`, `_FeedsBus` + type-only import) landed with the caps-sec slim + commit instead of the fixture-slim commit, keeping + each intermediate tree import-clean without + sub-hunk surgery. + +Also of record: the plan's "Verification" matrix was +executed as written (per-suite gates each stage, the +headless datad-feed smoke, the kucoin fail-fast unit +check); the known pre-existing ~50% second-runtime-boot +test wedge was characterized and excluded as a +regression via revert-testing.