Share piker skills across coding harnesses

Use portable Agent Skills metadata and one repo-local source for
both Claude Code and OpenCode.

Deats,
- replace harness-only frontmatter and dynamic context
- make `commit-msg` runtime-aware and review-first
- add a thin OpenCode command and integration docs
- guard `np.searchsorted()` guidance before OOB indexing
- record OpenCode prompt provenance

Prompt-IO: ai/prompt-io/opencode/20260721T205228Z_58ffe487_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))

.claude/settings.local.json

@@ -3,7 +3,8 @@
     "allow": [
       "Bash(chmod:*)",
       "Bash(/tmp/piker_commits.txt)",
-      "Bash(python:*)"
+      "Bash(python:*)",
+      "Bash(ls:*)"
     ],
     "deny": [],
     "ask": []

.claude/skills/commit-msg/SKILL.md

@@ -1,81 +1,116 @@
 ---
 name: commit-msg
 description: >
-  Generate piker-style git commit messages from
-  staged changes or prompt input, following the
-  style guide learned from 500 repo commits.
-argument-hint: "[optional-scope-or-description]"
-disable-model-invocation: true
-allowed-tools: Bash(git *), Read, Grep, Glob, Write
+  Generate piker-style git commit messages from staged
+  changes or prompt input. Use when the user asks for a
+  commit message or invokes a commit-message command.
+compatibility: >
+  Requires git and a coding harness able to read and
+  write files in the active checkout.
+metadata:
+  author: goodboy
+  version: "1.0"
 ---
 
-## Current staged changes
-!`git diff --staged --stat`
-
-## Recent commit style reference
-!`git log --oneline -10`
-
 # Piker Git Commit Message Generator
 
-Generate a commit message from the staged diff above
-following the piker project's conventions (learned from
-analyzing 500 repo commits).
-
-If `$ARGUMENTS` is provided, use it as scope or
-description context for the commit message.
+Generate a commit message from the staged diff following
+the piker project's conventions, learned from 500 repo
+commits.
 
 For the full style guide with verb frequencies,
 section markers, abbreviations, piker-specific terms,
 and examples, see
 [style-guide-reference.md](./style-guide-reference.md).
 
-## Quick Reference
+## Safety
+
+- Never run `git commit`, amend, rebase, merge, or push
+  unless the user explicitly requests that operation.
+- Do not stage or unstage files for a single-message run.
+- Never change issue, plan, or checklist state merely
+  because the related implementation is complete.
+- Preserve unrelated worktree changes.
+
+## Working Context
+
+1. Run `git rev-parse --show-toplevel`,
+   `git rev-parse --git-dir`, and
+   `git rev-parse --git-common-dir`.
+2. If the git dir differs from the common dir, report the
+   active worktree before generating the message.
+3. Inspect `git status --short`, the complete staged diff,
+   `git diff --staged --check`, and recent commit history.
+4. If nothing is staged, stop and tell the user. Prompt
+   text may clarify intent but does not replace a staged
+   patch unless the user explicitly asks for a draft with
+   no staged changes.
+
+## Message Style
 
 - **Subject**: ~50 chars, present tense verb, use
-  backticks for code refs
+  backticks for code refs; hard maximum 67 columns
 - **Body**: only for complex/multi-file changes,
-  67 char line max
+  filled close to a 67-column maximum
 - **Section markers**: Also, / Deats, / Other,
 - **Bullets**: use `-` style
 - **Tone**: technical but casual (piker style)
 
-## Claude-code Footer
+Use any user-supplied scope or description as context,
+but verify it against the staged patch. The message must
+describe the complete staged boundary, not only the most
+obvious file.
 
-When the written **patch** was assisted by
-claude-code, include:
+## Provenance
+
+If staged paths include prompt-I/O log entries under
+`ai/prompt-io/`, add one trailer per non-raw log file:
 
 ```
-(this patch was generated in some part by [`claude-code`][claude-code-gh])
-[claude-code-gh]: https://github.com/anthropics/claude-code
+Prompt-IO: ai/prompt-io/<service>/<entry>.md
 ```
 
-When only the **commit msg** was written by
-claude-code (human wrote the patch), use:
+When the patch has substantive AI-authored changes but no
+prompt-I/O entry is staged, remind the user that repository
+policy may require one. Do not block message generation.
+
+Always identify the active harness and model from runtime
+context rather than hardcoding Claude Code or OpenCode.
+
+When the harness assisted with the patch, append:
+
 ```
-(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
-[claude-code-gh]: https://github.com/anthropics/claude-code
+(this patch was generated in some part by `<harness>` using `<model>` (`<provider>`))
 ```
 
-## Output Instructions
+When it generated only the message, append:
 
-When generating a commit message:
+```
+(this commit msg was generated in some part by `<harness>` using `<model>` (`<provider>`))
+```
 
-1. Analyze the staged diff (injected above via
-   dynamic context) to understand all changes.
-2. If `$ARGUMENTS` provides a scope (e.g.,
-   `.ib.feed`) or description, incorporate it into
-   the subject line.
-3. Write the subject line following verb + backtick
-   conventions from the
-   [style guide](./style-guide-reference.md).
-4. Add body only for multi-file or complex changes.
-5. Write the message to a file in the repo's
-   `.claude/` subdir with filename format:
-   `<timestamp>_<first-7-chars-of-last-commit-hash>_commit_msg.md`
-   where `<timestamp>` is from `date --iso-8601=seconds`.
-   Also write a copy to
-   `.claude/git_commit_msg_LATEST.md`
-   (overwrite if exists).
+## Output Files
+
+Write the final message to both of these paths relative to
+the active checkout returned by `git rev-parse
+--show-toplevel`:
+
+- `.claude/skills/commit-msg/msgs/<timestamp>_<hash>_commit_msg.md`
+- `.claude/git_commit_msg_LATEST.md`
+
+Use UTC `<timestamp>` format `YYYYMMDDTHHMMSSZ` and the
+first seven characters of the current `HEAD` hash. Create
+the archive directory when needed. The `.claude` path is a
+legacy repository artifact location shared by all coding
+harnesses; it does not imply which harness generated the
+message.
+
+Finish by reporting both paths and this review-first
+command without running it:
+
+```
+git commit --edit --file .claude/git_commit_msg_LATEST.md
+```
 
 ---
 

.claude/skills/commit-msg/style-guide-reference.md

@@ -150,21 +150,18 @@ Common in piker commits (33.0% use colons):
 - File:line references not used (0 occurrences)
 - No WIP commits in analyzed set
 
-### Claude-code Footer
-When the written **patch** was assisted by claude-code,
-include:
+### Coding-harness Footer
+When a coding harness assisted with the written patch,
+identify the active harness, model, and provider:
 
 ```
-(this patch was generated in some part by [`claude-code`][claude-code-gh])
-[claude-code-gh]: https://github.com/anthropics/claude-code
+(this patch was generated in some part by `<harness>` using `<model>` (`<provider>`))
 ```
 
-When only the **commit msg** was written by claude-code
-(human wrote the patch), use:
+When it generated only the commit message, use:
 
 ```
-(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
-[claude-code-gh]: https://github.com/anthropics/claude-code
+(this commit msg was generated in some part by `<harness>` using `<model>` (`<provider>`))
 ```
 
 ## Piker-Specific Terms

.claude/skills/piker-conc-expert/SKILL.md

@@ -0,0 +1,155 @@
+---
+name: piker-conc-expert
+description: >
+  Distributed-runtime and structured-concurrency
+  expertise for piker's `tractor` actor-tree. Apply
+  when working on daemon/service architecture, actor
+  spawning/discovery, cross-actor RPC (ctx/stream
+  eps), `to_asyncio` integration, cancellation
+  semantics, or debugging hangs/wedges/skews in the
+  actor system.
+compatibility: >
+  Requires a piker checkout and familiarity with Python,
+  trio, and tractor.
+metadata:
+  author: goodboy
+  version: "1.0"
+---
+
+# Piker Concurrency & Runtime Expertise
+
+The distilled mental model for piker's distributed
+runtime: a `trio`-structured actor tree supervised by
+`tractor` (pinned to git main) where every long-lived
+subsystem is a named daemon-actor talking over
+ctx/stream IPC.
+
+## Actor tree & daemon taxonomy
+
+```
+pikerd                  root supervisor + registry
+├── datad.<broker>      feed bus, shm writers, tsp
+│                       history, symbol search
+├── brokerd.<broker>    live order-ctl ONLY; lazily
+│                       spawned by emsd, credentialed
+├── emsd                dark-clearing + order routing
+│   └── paperboi.<broker>  sim-clearing (paper mode)
+└── samplerd            singleton OHLC clock/increment
+```
+
+Key invariants:
+- `datad` hosts all `piker.data.validate._eps['datad']`
+  eps; `brokerd` only the `['brokerd']` (order-ctl)
+  ones. The `_eps` table in `piker/data/validate.py`
+  is the authoritative contract; `get_eps(mod, kind)`
+  introspects a backend's support.
+- `brokerd.<broker>` is booted in EXACTLY one place:
+  `open_brokerd_dialog()` in `piker/clearing/_ems.py`
+  (with a `portal:` override for the `piker ledger`
+  ad-hoc actor). Chart-only + paper sessions run with
+  ZERO brokerd procs. Never add a data-path spawn!
+- backends declare per-daemon-kind submods via
+  `_datad_mods`/`_brokerd_mods` in their
+  `__init__.py` (fallback: `__enable_modules__`).
+
+## Daemon lifecycle conventions
+
+Every daemon-kind follows the same trio of fns (see
+`piker/brokers/_daemon.py` + `piker/data/_daemon.py`
+as the canonical pair):
+
+- `_setup_persistent_<kind>()`: a `@tractor.context`
+  "lifetime fixture" run via
+  `Services.start_service_task()`; does console-log
+  setup ONCE for the actor, allocs any actor-global
+  state (eg. datad's `_FeedsBus`), then
+  `await ctx.started()` + `trio.sleep_forever()`.
+- `<kind>_init()`: builds `enable_modules` + actor
+  name `f'<kind>.{brokername}'` and copies backend
+  `_spawn_kwargs` (CRITICAL: `ib` needs
+  `infect_asyncio=True` in EVERY daemon-kind).
+- `spawn_<kind>()` + `maybe_spawn_<kind>()`: thin
+  wrappers over `Services.actor_n.start_actor()` and
+  `piker.service.maybe_spawn_daemon()` (registry
+  find-or-spawn w/ per-service-name locking).
+
+Caps-sec model: `enable_modules` gates RPC entry ONLY
+— python imports are unrestricted in-proc. Keep each
+daemon's enable set minimal; the (credentialed)
+`brokerd` must never RPC-enable `piker.data.*` feed
+mods.
+
+## Actor-local state: the #1 split hazard
+
+Module-globals and instance caches are PER-ACTOR.
+Anything that "just worked" because two subsystems
+shared a process will break when they're split into
+sibling actors. Canonical example: `ib`'s
+`Client._contracts` was warmed by feed-side
+`get_mkt_info()` in-proc; post datad/brokerd-split
+the trading actor must warm it itself (eagerly at
+`open_trade_dialog()` startup for open pps/orders +
+lazily per order request via
+`symbols.cache_contract()`).
+
+When moving code across actor boundaries ALWAYS audit:
+- module-global registries (`feed._bus`,
+  `_accounts2clients`, `_client_cache`, ..)
+- `@async_lifo_cache`/`maybe_open_context` caches
+  (NOTE: `async_lifo_cache` keys on POSITIONAL args
+  only; a cache-hit SKIPS the fn body and thus any
+  side-effect writes!)
+- logging handler placement (see gotchas.md)
+
+## tractor primitives as used here
+
+- `@tractor.context` eps: `await ctx.started(val)`
+  unblocks the caller w/ `val`; long-lived eps then
+  `ctx.open_stream()` or `sleep_forever()`.
+- discovery: `tractor.find_actor()` via
+  `piker.service.find_service()`;
+  `wait_for_actor(name, registry_addr=...)`;
+  `query_actor(name, regaddr=...)` yields
+  `(sockaddr, portal)`. Addrs are wrapped
+  `tractor.discovery._addr.Address` types — use
+  `wrap_address()` to normalize raw tuples and
+  `.unwrap()` for comparisons.
+- runtime-vars: `_runtime_vars['piker_vars']` is
+  inherited down the spawn tree; used eg. for
+  `piker_test_dir` config isolation — read LAZILY at
+  use-time, never at import time (subactors only get
+  vars post runtime-boot).
+- cancellation semantics (modern tractor): a
+  `ContextCancelled` whose `.canceller` is your own
+  actor is ABSORBED (clean exit, nothing raised);
+  single-exc groups collapse (`collapse_eg`) so eg.
+  a KBI propagates bare. Exc attrs:
+  `RemoteActorError.boxed_type` (not `.type`).
+
+## `to_asyncio` (infect-asyncio) integration
+
+For `ib` (and `deribit`) the backend client runs on
+an embedded `asyncio` loop via
+`tractor.to_asyncio.open_channel_from()` +
+`LinkedTaskChannel`.
+
+Rules learned the hard way:
+- a shared req/resp channel MUST correlate responses
+  to requests (see `MethodProxy._run_method()`'s
+  `mid` protocol in `piker/brokers/ib/api.py`):
+  caller cancellation (eg. `move_on_after` timeouts)
+  otherwise orphans a response and silently skews
+  every later result off-by-one.
+- the aio-side relay must catch + ship back ALL
+  (non-cancel) exceptions as `{'exception': err}`
+  resps; an escaping error kills the relay task ->
+  channel -> proxy nursery -> the whole dialog,
+  bypassing every caller-side guard.
+- `TrioTaskExited` ("child asyncio task is still
+  running?") on teardown is a known wart family;
+  prefer upstream `tractor` fixes over piker-side
+  bandaids.
+
+See [gotchas.md](gotchas.md) for the symptom->cause
+registry and [debug-recipes.md](debug-recipes.md) for
+forensics techniques.

.claude/skills/piker-conc-expert/debug-recipes.md

@@ -0,0 +1,100 @@
+# Debug recipes: actor-system forensics
+
+Field-tested techniques for diagnosing hangs, wedges
+and cross-actor state bugs WITHOUT a debugger attached
+(or when `py-spy` ain't installed).
+
+## Wedged actor triage (no REPL)
+
+1. find the tree:
+   `ps -eo pid,etime,args | grep -E 'pytest|tractor._child'`
+   — long-`etime` `tractor._child` procs w/ a stuck
+   parent = wedge.
+2. kernel state:
+   `cat /proc/<pid>/wchan` + `status | grep -E
+   'State|Threads'` — `do_epoll_wait` + sleeping =
+   idle event loop, NOT cpu-spin.
+3. **the money read** — socket queues:
+   `ss -tnp | grep <pid>`
+   - `Recv-Q > 0` on the parent-IPC conn = the actor
+     STOPPED CONSUMING its msg loop (runtime bug),
+     parent is waiting on it.
+   - zero external (api/ws) conns = wedged before/
+     without provider IO; don't blame the network.
+   - `CLOSE-WAIT` lingerers = unclean peer teardown.
+4. cleanup: `pkill -f tractor._child` (NB: in
+   compound shell cmds `pkill`'s exit code poisons
+   `&&` chains — run it standalone).
+
+## Hang-proof test gating
+
+- per-suite, never combined (cross-suite session
+  state interacts w/ the 2nd-boot wedge):
+  `timeout -k 5 300 python -m pytest tests/<one>.py -q`
+- rc 124/143 = hang-kill -> retry ONCE before
+  investigating.
+- isolate a flaky test w/ a 3x loop; ~50% hit-rate
+  signatures match the known 2nd-boot wedge (see
+  gotchas.md).
+
+## Regression vs pre-existing attribution
+
+When a failure appears mid-refactor:
+1. `git stash -u` (or checkout the file subset) and
+   re-run the EXACT failing case at baseline.
+2. if baseline can't even run, selectively revert
+   ONLY the suspect layer:
+   `git diff <files> > /tmp/x.patch;
+    git checkout <files>` -> test ->
+   `git apply /tmp/x.patch`.
+3. flake-rate compare (3x runs) beats single-shot
+   conclusions.
+
+## Off-by-one / stale IPC resp detection
+
+Mismatched query->result content in logs (resp
+payload obviously for a prior request) = shared
+req/resp channel w/o correlation + a cancelled
+caller. Grep the ep for `move_on_after`/`fail_after`
+around proxied calls. Fix = req-id (`mid`) tagging,
+never "just a lock" (cancellation still orphans).
+
+## Logging-chain audits
+
+When records double-print or go bare (see gotchas.md):
+
+```python
+import logging
+l = logging.getLogger('piker.brokers.ib.broker')
+while l:
+    print(l.name, l.level, l.handlers, l.propagate)
+    l = l.parent
+```
+
+Exactly ONE stderr handler should exist in the chain,
+attached by the actor's daemon fixture.
+
+## Live actor-tree smoke (headless)
+
+Boot against an ALT registry port so a user's running
+stack is untouched; script in a REAL file (tractor
+children re-exec `__main__` from path — stdin scripts
+crash w/ `FileNotFoundError: .../<stdin>`):
+
+```python
+async with maybe_open_pikerd(
+    registry_addrs=[('127.0.0.1', 6979)],
+):
+    async with open_feed(['xbtusdt.kraken']) as feed:
+        assert await check_for_service('datad.kraken')
+        assert not await check_for_service(
+            'brokerd.kraken'
+        )
+```
+
+## In-proc fail-fast unit checks
+
+Spawn-path guards that raise BEFORE touching the
+runtime can be tested w/ a bare `trio.run()` (eg
+`spawn_brokerd('kucoin')` raising the datad-only
+error) — no pikerd needed.

.claude/skills/piker-conc-expert/gotchas.md

@@ -0,0 +1,116 @@
+# Known gotchas: symptom -> cause -> fix
+
+A registry of distributed-runtime failure modes hit
+(and diagnosed) in the field; check here FIRST when a
+log/traceback matches.
+
+## "Can not order ..., no qualified contract cached"
+
+- **Symptom**: `RuntimeError` from
+  `ib.api.Client.submit_limit()` w/ empty
+  `Client._contracts` in `brokerd.ib`.
+- **Cause**: per-actor cache never warmed; feed-side
+  qualification now lives in `datad.ib`.
+- **Fix(ed)**: eager warmup at `open_trade_dialog()`
+  start + lazy per-order `get_mkt_info()` +
+  `cache_contract()` (writes BOTH `mkt.bs_fqme` and
+  `mkt.fqme` keys; different consumers read each!).
+
+## Search returns results for the WRONG pattern
+
+- **Symptom**: fqme search for 'gld' returns nvda
+  results; next query returns the prior query's set.
+- **Cause**: `MethodProxy` channel off-by-one — a
+  caller cancelled (search `move_on_after` timeout)
+  after sending its request orphans the response;
+  every later caller consumes the previous resp.
+- **Fix(ed)**: `mid` req-id correlation in
+  `_run_method()` + relay; stale resps are dropped w/
+  a "Dropping stale method-resp" warning. If that
+  warning spams, some caller is being cancelled
+  mid-call habitually — find + fix its timeout.
+
+## One bad request crashes a whole dialog/actor
+
+- **Symptom**: `TrioTaskExited` storm + nursery
+  teardown after a single method error (eg ambiguous
+  contract `AttributeError`).
+- **Cause**: exception escaped the aio-side relay
+  loop (`open_aio_client_method_relay()`) killing
+  channel + proxy nursery; caller-side `try/except`
+  CANNOT catch it.
+- **Fix(ed)**: relay catches `Exception` -> ships
+  `{'exception': err, 'mid': ...}` resp; order
+  handler converts to EMS `BrokerdError` msgs.
+
+## Ambiguous ib contracts -> `NoneType` attr errors
+
+- **Symptom**: `'NoneType' object has no attribute
+  'primaryExchange'` in `find_contracts()`.
+- **Cause**: `qualifyContractsAsync()` returns `None`
+  entries for ambiguous (eg venue-less stonk fqme
+  matching multiple listings: 'gld' -> ARCA/USD +
+  VENTURE/CAD).
+- **Fix(ed)**: filter `None`s + raise descriptive
+  `ValueError` ("use 'gld.arca.ib'").
+
+## Double-printed log records (same task id, 2x)
+
+- **Symptom**: every record from some subsys printed
+  twice w/ identical task ids.
+- **Cause**: stderr handlers attached at TWO levels
+  of one logger-propagation chain (eg daemon fixture
+  on `piker.brokers.ib` + an ep calling
+  `get_console_log(name=__name__)` on the child).
+  tractor's handler-dedup only checks the SAME
+  logger, not ancestors.
+- **Rule**: console handlers are attached ONCE per
+  actor in the `_setup_persistent_*()` fixture; eps
+  needing a different level use `log.setLevel()`
+  ONLY, never `get_console_log()`.
+
+## Bare/non-colorized log lines
+
+- **Symptom**: records w/ no timestamp/actor prefix.
+- **Cause**: NO handler anywhere in the emitting
+  logger's chain -> stdlib `logging.lastResort`. Post
+  actor-splits, a daemon fixture may only cover its
+  own subsys subtree (eg datad's `piker.data.*` but
+  not the backend's `piker.brokers.<broker>.*`).
+- **Fix(ed)**: `_setup_persistent_datad()` enables
+  BOTH `piker.data.<broker>` and
+  `piker.brokers.<broker>` subtrees.
+
+## 2nd in-proc runtime boot wedges (~50%)
+
+- **Symptom**: test hangs when one test proc boots a
+  2nd `pikerd` (eg `test_multi_fill_positions`'s
+  persistence re-check); a zombie `*.{broker}` child
+  lingers w/ unread bytes in its parent-IPC Recv-Q.
+- **Cause**: pre-existing `tractor`-main runtime
+  teardown bug (confirmed independent of piker-layer
+  changes via revert-testing 2026-06).
+- **Mitigation**: run suites per-file wrapped in
+  `timeout -k 5 300 ...`; retry once on rc 124/143.
+  Do NOT chase as a regression of unrelated changes.
+
+## ib client-id collisions post-split
+
+- **Symptom**: 2nd ib daemon burns the full
+  conn-timeout retry cycle connecting to gw/tws.
+- **Cause**: `datad.ib` + `brokerd.ib` both default
+  `client_id=6116` w/ linear `+i` retries.
+- **Fix(ed)**: role-based offsets in
+  `load_aio_clients()`: datad +16, ad-hoc (test/cli)
+  actors +32.
+
+## `async_lifo_cache` skipped side-effects
+
+- **Symptom**: a fn's cache-write side effect
+  (eg `get_mkt_info()` -> `_contracts`) missing for
+  a 2nd client/proxy.
+- **Cause**: cache keys on POSITIONAL args only; a
+  hit skips the body entirely.
+- **Rule**: never rely on cached-fn side effects;
+  perform required writes explicitly at the call
+  site (eg `cache_contract()` after `get_mkt_info`).

.claude/skills/piker-profiling/SKILL.md

@@ -5,7 +5,11 @@ description: >
   across distributed actor systems. Apply when
   adding profiling, debugging perf regressions, or
   optimizing hot paths in piker code.
-user-invocable: false
+compatibility: >
+  Requires a piker checkout and Python profiling work.
+metadata:
+  author: goodboy
+  version: "1.0"
 ---
 
 # Piker Profiling Subsystem

.claude/skills/piker-slang/SKILL.md

@@ -5,7 +5,11 @@ description: >
   ethos. Apply when communicating with piker devs,
   writing commit messages, code review comments, or
   any collaborative interaction.
-user-invocable: false
+compatibility: >
+  Harness-independent communication guidance.
+metadata:
+  author: goodboy
+  version: "1.0"
 ---
 
 # Piker Slang & Communication Style

.claude/skills/pyqtgraph-optimization/SKILL.md

@@ -5,7 +5,12 @@ description: >
   for piker's UI. Apply when optimizing graphics
   performance, adding new chart annotations, or
   working with `QGraphicsItem` subclasses.
-user-invocable: false
+compatibility: >
+  Requires a piker checkout with PyQtGraph and Qt source
+  available for inspection.
+metadata:
+  author: goodboy
+  version: "1.0"
 ---
 
 # PyQtGraph Rendering Optimization

.claude/skills/timeseries-optimization/SKILL.md

@@ -6,7 +6,12 @@ description: >
   with OHLCV arrays, timestamp lookups, gap
   detection, or any array/dataframe operations in
   piker.
-user-invocable: false
+compatibility: >
+  Requires a piker checkout with NumPy and optionally
+  Polars.
+metadata:
+  author: goodboy
+  version: "1.0"
 ---
 
 # Timeseries Optimization: NumPy & Polars
@@ -61,11 +66,14 @@ ts_array = np.array(timestamps)
 # binary search for all timestamps at once
 indices = np.searchsorted(time_arr, ts_array)
 
-# bounds check and exact match verification
-valid_mask = (
-    (indices < len(array))
-    &
-    (time_arr[indices] == ts_array)
+# bounds check before indexing: searchsorted may return
+# len(time_arr) for values above the final timestamp
+in_bounds = indices < len(time_arr)
+valid_mask = np.zeros(indices.shape, dtype=bool)
+valid_mask[in_bounds] = (
+    time_arr[indices[in_bounds]]
+    ==
+    ts_array[in_bounds]
 )
 
 valid_indices = indices[valid_mask]

.opencode/commands/commit-msg.md

@@ -0,0 +1,10 @@
+---
+description: Generate a piker-style message for staged changes.
+---
+
+Load the `commit-msg` skill and follow it completely for the currently
+staged changes. Do not run `git commit` unless the user explicitly asks.
+Keep all workflow and style decisions in the shared skill rather than
+duplicating them in this OpenCode command.
+
+Additional context from the user: $ARGUMENTS

ai/README.md

@@ -17,6 +17,18 @@ track new integration ideas and proposals in
 | Tool | Directory | Status |
 |------|-----------|--------|
 | [Claude Code](https://github.com/anthropics/claude-code) | [`claude-code/`](claude-code/) | active |
+| [OpenCode](https://opencode.ai/) | [`opencode/`](opencode/) | active |
+
+## Shared Skills
+
+Repo-specific skills use the portable Agent Skills
+frontmatter subset and live under `.claude/skills/` as a
+single source. Claude Code discovers that directory
+natively, and OpenCode discovers the same project skills
+without copies or generated wrappers.
+
+Harness-specific command files should only delegate to a
+shared skill. They must not duplicate the skill body.
 
 ## Adding a New Integration
 
@@ -32,7 +44,7 @@ ai/
 ├── README.md                  # <- you are here
 ├── claude-code/
 │   └── README.md
-├── opencode/                  # future
+├── opencode/
 │   └── README.md
 └── <your-tool>/
     └── README.md
@@ -45,6 +57,7 @@ ai/
 - Each integration doc should describe **what**
   the skill does, **how** to invoke it, and any
   **output** artifacts it produces
-- Keep docs concise; link to the actual skill
-  source files (under `.claude/skills/`, etc.)
+- Keep docs concise; link to the shared skill source files
   rather than duplicating content
+- Keep skill bodies harness-neutral; isolate slash-command
+  syntax and permissions in each harness integration

ai/claude-code/README.md

@@ -1,15 +1,17 @@
 # Claude Code Integration
 
 [Claude Code](https://github.com/anthropics/claude-code)
-skills and workflows for piker development.
+integration for piker's shared coding-harness skills.
 
 ## Skills
 
 | Skill | Invocable | Description |
 |-------|-----------|-------------|
 | [`commit-msg`](#commit-msg) | `/commit-msg` | Generate piker-style commit messages |
+| `piker-conc-expert` | auto | Actor-tree and structured-concurrency invariants |
 | `piker-profiling` | auto | `Profiler` API patterns for perf work |
 | `piker-slang` | auto | Communication style + slang guide |
+| `py-codestyle` | auto | Python source conventions |
 | `pyqtgraph-optimization` | auto | Batch rendering patterns |
 | `timeseries-optimization` | auto | NumPy/Polars perf patterns |
 
@@ -17,8 +19,10 @@ Skills marked **auto** are background knowledge
 applied automatically when Claude detects relevance.
 Only `commit-msg` is user-invoked via slash command.
 
-Skill source files live under
-`.claude/skills/<skill-name>/SKILL.md`.
+Portable skill source files live under
+`.claude/skills/<skill-name>/SKILL.md` and are shared with
+OpenCode. Claude-specific behavior belongs in command or
+settings files, not the shared skill bodies.
 
 ---
 
@@ -84,8 +88,7 @@ description context. Examples:
 **Footer** (always):
 ```
 (this patch was generated in some part by
-[`claude-code`][claude-code-gh])
-[claude-code-gh]: https://github.com/anthropics/claude-code
+`claude-code` using `<model>` (`<provider>`))
 ```
 
 ### Output Files
@@ -94,7 +97,8 @@ After generation, the commit message is written to:
 
 ```
 .claude/
-├── <timestamp>_<hash>_commit_msg.md   # archived
+├── skills/commit-msg/msgs/
+│   └── <timestamp>_<hash>_commit_msg.md
 └── git_commit_msg_LATEST.md           # latest
 ```
 
@@ -102,18 +106,10 @@ Where `<timestamp>` is ISO-8601 with seconds and
 `<hash>` is the first 7 chars of the current
 `HEAD` commit.
 
-Use the latest file to feed into `git commit`:
+Use the latest file with an editor review before commit:
 
 ```bash
-git commit -F .claude/git_commit_msg_LATEST.md
-```
-
-Or review/edit before committing:
-
-```bash
-cat .claude/git_commit_msg_LATEST.md
-# edit if needed, then:
-git commit -F .claude/git_commit_msg_LATEST.md
+git commit --edit --file .claude/git_commit_msg_LATEST.md
 ```
 
 ### Examples
@@ -128,56 +124,32 @@ Add `MktPair.fqme` property for symbol resolution
 Factor `.claude/skills/` into proper subdirs
 
 Deats,
-- `commit_msg/` -> `commit-msg/` w/ enhanced
-  frontmatter
-- all background skills set `user-invocable: false`
-- content split into supporting files
+- use portable Agent Skills frontmatter
+- keep harness commands as thin delegates
+- share supporting references across harnesses
 
 (this patch was generated in some part by
-[`claude-code`][claude-code-gh])
-[claude-code-gh]: https://github.com/anthropics/claude-code
+`claude-code` using `<model>` (`<provider>`))
 ```
 
-### Frontmatter Reference
+### Shared Frontmatter
 
-The skill's `SKILL.md` uses these Claude Code
-frontmatter fields:
+Skills use the Agent Skills fields understood by both
+Claude Code and OpenCode:
 
 ```yaml
 ---
 name: commit-msg
 description: >
-  Generate piker-style git commit messages...
-argument-hint: "[optional-scope-or-description]"
-disable-model-invocation: true
-allowed-tools:
-  - Bash(git *)
-  - Read
-  - Grep
-  - Glob
-  - Write
+  Generate piker-style git commit messages. Use when...
+compatibility: Requires git.
+metadata:
+  author: goodboy
+  version: "1.0"
 ---
 ```
 
-| Field | Purpose |
-|-------|---------|
-| `argument-hint` | Shows hint in autocomplete |
-| `disable-model-invocation` | Only user can trigger via `/commit-msg` |
-| `allowed-tools` | Tools the skill can use |
-
-### Dynamic Context
-
-The skill injects live data at invocation time
-via `!`backtick`` syntax in the `SKILL.md`:
-
-```markdown
-## Current staged changes
-!`git diff --staged --stat`
-
-## Recent commit style reference
-!`git log --oneline -10`
-```
-
-This means the staged diff stats and recent log
-are always fresh when the skill runs -- no stale
-context.
+Live git context is gathered by the skill at execution
+time. Claude-only dynamic interpolation and tool-policy
+frontmatter are intentionally excluded so another harness
+can execute the same workflow.

ai/claude-code/plans/datad_service.md

@@ -0,0 +1,143 @@
+# Split `brokerd.<broker>` into trading-only `brokerd` + new `datad.<broker>`
+
+## Context
+
+Today a single `brokerd.<broker>` actor hosts BOTH concerns:
+
+- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers/<backend>/feed.py`/`symbols.py`,
+- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers/<backend>/broker.py`, driven by `emsd`.
+
+The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split.
+
+**User-decided constraints:**
+
+- `datad.<broker>` is a **sibling** of `brokerd.<broker>` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`).
+- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working.
+- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes.
+
+## Target topology
+
+```
+pikerd
+├── datad.ib      feed bus, shm writers, tsp history, symbol search
+├── brokerd.ib    open_trade_dialog only (EMS-spawned, lazy)
+├── emsd
+│   └── paperboi.ib   (paper mode; opens its feed via datad)
+└── samplerd
+```
+
+## Key verified facts (load-bearing)
+
+- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor.
+- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree.
+- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param.
+- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS.
+- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns).
+- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`.
+- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split.
+- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves.
+- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad.
+
+## Module decision
+
+New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication).
+
+---
+
+## Stage 0 — prep: backend module grouping + ep introspection (no topology change)
+
+1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks.
+2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged:
+   - `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']`
+   - `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']`
+   - `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']`
+   - `ib`: adjust existing groups so each includes `'api'`
+   - flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior).
+3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`).
+
+**Gate**: full pytest green, zero behavior change.
+
+## Stage 1 — introduce `datad` daemon machinery (additive)
+
+New `piker/data/_daemon.py`:
+
+- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`.
+- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`.
+- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`.
+- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`.
+- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`).
+
+Supporting edits:
+
+- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`).
+- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`).
+- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.)
+
+**Gate**: suite green + new spawn test; nothing routes through datad yet.
+
+## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`)
+
+Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal).
+
+`piker/clearing/_ems.py`:
+
+1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it.
+2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`).
+3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`.
+
+Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry.
+
+**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke.
+
+## Stage 3 — feed layer hard cutover to datad
+
+1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep.
+2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.)
+3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`:
+   - `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`)
+   - `piker/brokers/core.py:33,175` (`symbol_search`)
+   - `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`)
+   - `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'`
+4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`.
+
+**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.<broker>` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane.
+
+## Stage 4 — caps-sec slimming, validation, ib client-id
+
+1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all.
+2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously.
+3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys.
+4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring.
+
+**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`.
+
+---
+
+## Risk register
+
+| Risk | Sev | Mitigation |
+|---|---|---|
+| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns |
+| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test |
+| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast |
+| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only |
+| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it |
+| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly |
+
+## Verification (per stage gates above, plus end-to-end)
+
+- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed).
+- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger <broker>.paper`.
+- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4).
+
+## Critical files
+
+- **new** `piker/data/_daemon.py` — all datad machinery
+- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn
+- `piker/data/feed.py` — spawn cutover + actor-name assert
+- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`)
+- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports
+- `piker/data/validate.py` — `get_eps()` helper
+- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods`
+- `piker/accounting/cli.py` — keyword-form dialog call
+- `tests/test_services.py` — inverted + new assertions

ai/opencode/README.md

@@ -0,0 +1,51 @@
+# OpenCode Integration
+
+[OpenCode](https://opencode.ai/) uses piker's shared
+repo-specific skills directly from `.claude/skills/`.
+There is no second copy of each skill to keep in sync.
+
+## Available Skills
+
+| Skill | Activation | Specialization |
+|-------|------------|----------------|
+| `commit-msg` | `/commit-msg` or explicit request | Piker commit-message style and artifacts |
+| `piker-conc-expert` | automatic | `tractor` actor topology, RPC, and cancellation |
+| `piker-profiling` | automatic | Cross-actor `Profiler` instrumentation |
+| `piker-slang` | automatic | Project communication style |
+| `py-codestyle` | automatic | Python source conventions |
+| `pyqtgraph-optimization` | automatic | Batched Qt graphics rendering |
+| `timeseries-optimization` | automatic | NumPy and Polars timeseries performance |
+
+OpenCode's project command lives at
+`.opencode/commands/commit-msg.md`. It delegates to the
+shared `commit-msg` skill and only supplies command
+arguments; workflow logic remains in the skill.
+
+A user-local skill with the same name may override a repo
+skill. Treat `opencode debug skill` as authoritative when
+diagnosing which source is active.
+
+## Verify Discovery
+
+From the repository root:
+
+```console
+opencode debug skill
+opencode debug config
+```
+
+`opencode debug skill` should list the piker skills from
+the project checkout. Restart OpenCode after changing a
+skill or command because configuration-time files are not
+hot-reloaded into an active session.
+
+## Portability Rules
+
+- Use only portable Agent Skills frontmatter in shared
+  `SKILL.md` files.
+- Keep harness-specific command syntax under
+  `.opencode/commands/`.
+- Detect the active harness and model at runtime; do not
+  hardcode Claude Code attribution in shared workflows.
+- Keep supporting references beside each shared skill so
+  relative links resolve in every harness.

ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md

@@ -0,0 +1,59 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:08:59Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
+---
+
+## Prompt
+
+Session-initiating instruction (driving all 7 commits in
+this series):
+
+> ok i want you to become the distributed runtime and
+> concurrency expert for this project - namely acquire
+> a deep understanding of tractor and how it's used.
+> then i want you to attempt to factor our current
+> brokerd service daemon into 2 daemons: a brokerd
+> which only hosts live/paper trading endpoint tasks
+> [..] a new `datad` subdaemon which in a separate
+> subactor serves all the data feed service tasks [..]
+> give me a mega detailed plan on how to approach this,
+> and a staged approach for the implementation.
+
+Proximate driver for THIS commit: during the approved
+plan's stage-0 test gate the agent discovered test
+subactors were writing into the user's REAL
+`~/.config/piker/accounting/` files (a bogus paper fill
+landed in `account.kraken.paper.toml`) because the old
+test-dir override in `config.get_app_dir()` was
+commented out and could never work in spawned
+subactors anyway. Fix applied autonomously under the
+approved plan; no explicit per-fix user prompt.
+
+## Response summary
+
+Restore `pytest` config-dir isolation by resolving the
+per-test tmp dir lazily (at conf-path access time) from
+`tractor` runtime-vars, which propagate down the actor
+tree; route all conf-path derivation through
+`config.get_conf_dir()` so the override is effective in
+every (sub)actor.
+
+## Files changed
+
+- `piker/config.py` — add `_maybe_use_test_dir()`;
+  hook in `get_conf_dir()`; route `get_conf_path()` +
+  `load()` mkdir through it
+- `piker/accounting/_ledger.py` — derive ledger dir
+  via `config.get_conf_dir()` (not the global)
+- `piker/accounting/_pos.py` — same for account files
+
+## Human edits
+
+None — committed as generated.

ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.raw.md

@@ -0,0 +1,46 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:08:59Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); backfilled from the live dev
+session transcript per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/config.py`
+
+Generated: `config._maybe_use_test_dir()` — lazily
+reads `piker_test_dir` from
+`tractor.runtime._state._runtime_vars['piker_vars']`
+(pre-loaded by `open_piker_runtime()` from the
+`tests.conftest._open_test_pikerd()` overrides) and
+calls `_override_config_dir()` when set. Hooked at the
+top of `get_conf_dir()`; `get_conf_path()` and
+`load()`'s dir-creation rerouted through
+`get_conf_dir()`.
+
+> `git log -1 -p --follow -- piker/accounting/_ledger.py`
+> `git log -1 -p --follow -- piker/accounting/_pos.py`
+
+Generated: ledger/account dir derivation switched from
+the `config._config_dir` module global to
+`config.get_conf_dir()` + `mkdir(parents=True,
+exist_ok=True)` for nested tmp-dir creation.
+
+Key diagnostic reasoning (verbatim from session):
+
+The old (commented-out) `get_app_dir()` override gated
+on `'pytest' in sys.modules` which can NEVER work in
+spawned subactors (fresh procs, no pytest import); as
+a result test `paperboi`/daemon actors were writing
+into the user's REAL `~/.config/piker/accounting/`
+files. Evidence: `account.kraken.paper.toml` gained a
+single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
+(a test fill), and `trades_kraken_paper.toml` (252B)
+contained only that fill. The resolution must be
+checked lazily at config-path access time (NOT import
+time) since sub-actors only receive runtime-vars once
+their `tractor` runtime has fully booted.

ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md

@@ -0,0 +1,53 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:10:22Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
+---
+
+## Prompt
+
+Same session-initiating `brokerd`-split instruction (see
+`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
+driver: the approved plan's per-stage test gates could
+not run AT ALL — the branch base was broken vs.
+`tractor` git `main` (`AttributeError: 'tuple' object
+has no attribute 'unwrap'` at `pikerd` boot, stale
+`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
+fixed forward autonomously to (re)establish the gate
+baseline, continuing the `repair_tests` branch lineage
+(verified already merged in ancestry via `git cherry`
+during a user-requested branch-overlap survey).
+
+## Response summary
+
+Port the service layer + test suites to current
+`tractor` APIs: addr-type normalization in
+`open_pikerd()`, `query_actor()`/`wait_for_actor()`
+kwarg renames, modern self-cancel absorption semantics
+in the cancel-method test harness, exc/position attr
+renames, a paper-EMS startup-budget bump and a syntax
+fix in `.deribit.api`.
+
+## Files changed
+
+- `piker/service/_actor_runtime.py` — `wrap_address()`
+  normalize before `.unwrap()` in `open_pikerd()`
+- `piker/service/_registry.py` — `check_for_service()`
+  -> `query_actor(regaddr=)` + 2-tuple yield +
+  `open_registry(addrs=)`
+- `piker/brokers/deribit/api.py` — missing comma in
+  `tractor.trionics` import tuple
+- `tests/test_services.py` — `registry_addr=` kwarg,
+  raddr unwraps, cancel-semantics harness rewrite,
+  `fail_after` 9 -> 19s
+- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
+
+## Human edits
+
+None — committed as generated.

ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.raw.md

@@ -0,0 +1,56 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:10:22Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); backfilled from the live dev
+session transcript per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
+
+Generated: normalize each registry addr via
+`tractor.discovery._addr.wrap_address()` before
+`.unwrap()`-ing for the `accept_addrs` bind check —
+entries may be raw `tuple`s when passed in from (test)
+client code. Import-path precedent taken from
+`piker/cli/__init__.py:336`.
+
+> `git log -1 -p --follow -- piker/service/_registry.py`
+
+Generated: `check_for_service()` ported to
+`tractor.query_actor(name, regaddr=...)` (kwarg was
+`arbiter_sockaddr=`), unpacking the new
+`(sockaddr, portal)` yield, and passing the
+now-required `open_registry(addrs=Registry.addrs)`.
+
+> `git log -1 -p --follow -- tests/test_services.py`
+> `git log -1 -p --follow -- tests/test_ems.py`
+> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
+
+Key diagnostic reasoning (verbatim from session):
+
+- the "DID NOT RAISE ContextCancelled" failure: in this
+  test the client actor IS pikerd (in-proc), and
+  current `tractor` main absorbs a `ContextCancelled`
+  whose canceller is your own actor — self-requested
+  cancels now exit cleanly instead of raising; the
+  'sigint' variant propagates a bare collapsed
+  `KeyboardInterrupt` rather than a
+  `BaseExceptionGroup`.
+- the hard-coded `trio.fail_after(9)` startup budget is
+  marginal — full stack boot (pikerd -> emsd ->
+  brokerd.kraken -> paperboi + live kraken symbology
+  fetch) occasionally exceeds 9s -> bumped to 19s.
+- `RemoteActorError.type` -> `.boxed_type`;
+  `Position.size` -> `.cumsize` (the paper engine
+  populates `BrokerdPosition.size` from `pp.cumsize`).
+- overlap survey (user-requested): all of the
+  `repair_tests` branch commits are already in this
+  stack's ancestry; this commit finishes that branch's
+  port mission (its f4c4f1e2 fixed `conftest.py`'s
+  `arbiter_sockaddr` usage; this fixes the remaining
+  `test_services.py` + `check_for_service()` sites).

ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md

@@ -0,0 +1,54 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:11:05Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T171105Z_bc6e18d7_prompt_io.raw.md
+---
+
+## Prompt
+
+Same session-initiating `brokerd`-split instruction (see
+`20260610T170859Z_75cefe10_prompt_io.md`); this is the
+approved plan's "stage 0" prep. Plan-shaping user
+decisions captured via in-session Q&A:
+
+- `datad.<broker>` topology: sibling of
+  `brokerd.<broker>` under `pikerd` (vs. child
+  subactor).
+- migration: hard cutover, staged by layer (no
+  dual-mode runtime flag).
+- post-split `brokerd` scope: trading-only +
+  EMS-lazy-spawned (charts/CLI never touch it).
+
+## Response summary
+
+Declare per-daemon-kind backend submod groups
+(`_datad_mods`/`_brokerd_mods`) in the split-style
+backends, keyed to the pre-existing
+`piker.data.validate._eps` contract; add a
+`validate.get_eps()` introspection helper; ws-import
+hygiene in `.kraken.broker`. Zero behavior change
+(`__enable_modules__` unions unchanged).
+
+## Files changed
+
+- `piker/data/validate.py` — add `get_eps()`
+- `piker/brokers/kraken/__init__.py` — mod groups
+- `piker/brokers/binance/__init__.py` — mod groups
+  (note: no `symbols.py`; search eps live in `.feed`)
+- `piker/brokers/deribit/__init__.py` — datad-only
+  groups (no `broker.py` yet)
+- `piker/brokers/ib/__init__.py` — add `'api'` to the
+  pre-existing `_datad_mods`
+- `piker/brokers/kraken/broker.py` — import
+  `NoBsWs`/`open_autorecon_ws` from
+  `piker.data._web_bs` directly
+
+## Human edits
+
+None — committed as generated.

ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.raw.md

@@ -0,0 +1,46 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:11:05Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); backfilled from the live dev
+session transcript per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/data/validate.py`
+
+Generated: `get_eps(mod, kind) -> dict[str, Callable]`
+returning the daemon-kind's ep funcs defined by a
+backend mod, keyed by ep name, missing eps excluded;
+sourced from the existing `_eps` grouping table
+(`'middleware' | 'datad' | 'brokerd'`).
+
+> `git log -1 -p --follow -- piker/brokers/kraken/__init__.py`
+> `git log -1 -p --follow -- piker/brokers/binance/__init__.py`
+> `git log -1 -p --follow -- piker/brokers/deribit/__init__.py`
+> `git log -1 -p --follow -- piker/brokers/ib/__init__.py`
+> `git log -1 -p --follow -- piker/brokers/kraken/broker.py`
+
+Key exploration findings driving the design (from the
+planning phase, 3 parallel explore agents + 1 architect
+agent):
+
+- the codebase already anticipated this split:
+  `validate._eps` groups eps into 'datad' vs 'brokerd'
+  kinds and `ib/__init__.py` already declared
+  `_brokerd_mods`/`_datad_mods`; `brokers/_daemon.py`
+  carried the literal TODO "rename the daemon to datad
+  prolly once we split up broker vs. data tasks into
+  separate actors?".
+- the feared kraken feed<->broker "shared ws state"
+  hazard is mild: `NoBsWs`/`open_autorecon_ws`
+  originate in `piker.data._web_bs` (re-exported via
+  `.kraken.feed`); only `stream_messages` is feed-local
+  and it's a pure parser — in-process imports only,
+  each actor opens its own ws conn.
+- `enable_modules` gates RPC entry, NOT python imports,
+  so backend trading mods may keep importing feed-side
+  helpers in-process post-split.

ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md

@@ -0,0 +1,47 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:11:42Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T171142Z_119d2c04_prompt_io.raw.md
+---
+
+## Prompt
+
+Same session-initiating `brokerd`-split instruction (see
+`20260610T170859Z_75cefe10_prompt_io.md`); this is the
+approved plan's "stage 1": introduce the `datad` daemon
+machinery additively (nothing routes through it yet).
+User-decided constraint applied: `datad.<broker>` is a
+SIBLING of `brokerd.<broker>` under `pikerd`, spawned
+via the existing `Services` + `maybe_spawn_daemon()`
+machinery.
+
+## Response summary
+
+New `piker/data/_daemon.py` hosting the
+`datad.<broker>` feed-only daemon-actor: lifetime
+fixture owning the actor-global `_FeedsBus`, init/spawn
+fns mirroring `.brokers._daemon` conventions and the
+`samplerd` sub-daemon precedent, plus root-mod
+registration, `piker.service` re-exports and a spawn
+test.
+
+## Files changed
+
+- `piker/data/_daemon.py` — NEW:
+  `_setup_persistent_datad()`, `datad_init()`,
+  `spawn_datad()`, `maybe_spawn_datad()`,
+  `_datad_service_mods`
+- `piker/service/_actor_runtime.py` — add
+  `piker.data._daemon` to `_root_modules`
+- `piker/service/__init__.py` — re-export spawn eps
+- `tests/test_services.py` — add `test_datad_spawn`
+
+## Human edits
+
+None — committed as generated.

ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.raw.md

@@ -0,0 +1,60 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:11:42Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); backfilled from the live dev
+session transcript per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/data/_daemon.py`
+
+Generated symbols + key design decisions:
+
+- `_datad_service_mods: list[str]` — datad-always
+  enabled mods, the data-side successor to the old
+  `piker.brokers._daemon._data_mods` set; kept minimal
+  per the caps-sec model.
+- `_setup_persistent_datad()` — `@tractor.context`
+  lifetime fixture: console-log setup then allocates
+  the actor-global feed bus via
+  `feed.get_feed_bus(brokername, service_nursery)`
+  exactly as the old brokerd fixture did, pinned open
+  with `ctx.started()` + `sleep_forever()`.
+- `datad_init()` — actor name `f'datad.{brokername}'`;
+  copies backend `_spawn_kwargs` (CRITICAL for `ib`'s
+  `infect_asyncio=True`); builds `enable_modules` from
+  `getattr(brokermod, '_datad_mods',
+  getattr(brokermod, '__enable_modules__', []))` as
+  the flat-backend fallback.
+- `spawn_datad()` — `Services.actor_n.start_actor()` +
+  `Services.start_service_task()` exactly mirroring
+  `spawn_brokerd()`; dedup-composes enable mods via
+  `list(dict.fromkeys(...))`.
+- `maybe_spawn_datad()` — wraps `maybe_spawn_daemon(
+  service_name=f'datad.{brokername}', ...)`.
+
+> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
+> `git log -1 -p --follow -- piker/service/__init__.py`
+> `git log -1 -p --follow -- tests/test_services.py`
+
+Design rationale (verbatim from session):
+
+- `_root_modules` must gain `piker.data._daemon` so
+  `pikerd_portal.run(spawn_datad, ...)` resolves in
+  the root.
+- the `Services`-based impl style deliberately mirrors
+  `spawn_brokerd()` so the eventual `tractor.hilevel`
+  `ServiceMngr` port (see the `service_mng_to_tractor`
+  branch's d8c21d44 prep, surfaced by the
+  user-requested branch-overlap survey) lands
+  symmetrically on both spawn fns.
+- mod placement (`piker/data/_daemon.py` vs.
+  generalizing `piker.brokers._daemon`) follows the
+  per-subsystem daemon-mod convention
+  (`.clearing._ems`, `.data._sampling`) and resolves
+  the existing TODO at `brokers/_daemon.py:49` ("move
+  this def to the `.data` subpkg").

ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md

@@ -0,0 +1,47 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:12:26Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T171226Z_64181219_prompt_io.raw.md
+---
+
+## Prompt
+
+Same session-initiating `brokerd`-split instruction (see
+`20260610T170859Z_75cefe10_prompt_io.md`); this is the
+approved plan's "stage 2": decouple the clearing layer
+from `feed.portals` BEFORE the feed cutover so live
+trading works at every stage boundary. User-decided
+constraint applied: post-split `brokerd` is
+trading-only and spawned LAZILY only by `emsd`'s
+`open_brokerd_dialog()` path.
+
+## Response summary
+
+Kill the single coupling forcing feed + trading eps
+into one actor (`Router.open_trade_relays()` pulling
+its trades portal from `feed.portals[brokermod]`):
+`open_brokerd_dialog()` now (maybe) spawns/finds
+`brokerd.<broker>` itself and ONLY when a live
+trades-ep will actually open; paper mode never touches
+it. Pre-cutover this is a pure refactor (registry
+lookup finds the same feed-spawned daemon).
+
+## Files changed
+
+- `piker/clearing/_ems.py` — `open_brokerd_dialog()`
+  re-sig + inner `acquire_live_portal()`;
+  `Router.maybe_open_brokerd_dialog()` drops `portal`
+  param; `open_trade_relays()` drops the
+  `feed.portals` lookup
+- `piker/accounting/cli.py` — keyword-form `portal=`
+  override kept for the `piker ledger` ad-hoc actor
+
+## Human edits
+
+None — committed as generated.

ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.raw.md

@@ -0,0 +1,43 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:12:26Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); backfilled from the live dev
+session transcript per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/clearing/_ems.py`
+
+Generated: new `open_brokerd_dialog()` signature
+`(brokermod, exec_mode, fqme=None, portal=None,
+loglevel=None)` with an inner `@acm
+acquire_live_portal()` that yields the caller-provided
+`portal` override (the `piker ledger` path) else
+`maybe_spawn_brokerd(brokermod.name)` — designated THE
+one place a live, credentialed `brokerd.<broker>` gets
+booted post-split. The eager
+`portal.open_context(trades_endpoint, ...)`
+construction moved inside that block so the paper
+short-circuit never acquires a live portal.
+
+> `git log -1 -p --follow -- piker/accounting/cli.py`
+
+Key analysis (verbatim from session):
+
+- `feed.portals` had exactly ONE trading consumer:
+  `piker/clearing/_ems.py:671` (`portal =
+  feed.portals[brokermod]`) — the single coupling
+  forcing feed + trading into one actor.
+- `piker ledger` (`accounting/cli.py:100-157`) is a
+  hidden consumer: it calls `broker_init()` directly,
+  spawns its own ad-hoc actor and passes the portal
+  positionally — the new signature must keep an
+  explicit `portal:` override param for this path.
+- stage sequencing: clearing decouple lands BEFORE the
+  feed cutover, otherwise `feed.portals` would hand
+  emsd a `datad` portal and live `open_trade_dialog`
+  RPC would fail.

ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md

@@ -0,0 +1,48 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:12:59Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T171259Z_59d5d9a6_prompt_io.raw.md
+---
+
+## Prompt
+
+Same session-initiating `brokerd`-split instruction (see
+`20260610T170859Z_75cefe10_prompt_io.md`); this is the
+approved plan's "stage 3": the feed-layer hard cutover
+(user-decided: no dual-mode runtime flag, each stage
+lands fully working).
+
+## Response summary
+
+The topology flip: all data-feed consumers route to
+`datad.<broker>`; the `brokerd` lifetime fixture slims
+to trading-only; UI/CLI data-flavoured spawn sites
+repointed; test expectations inverted incl. an explicit
+negative assert that paper sessions spawn NO
+`brokerd.<broker>`.
+
+## Files changed
+
+- `piker/data/feed.py` — `open_feed()` ->
+  `maybe_spawn_datad()`; `open_feed_bus()` actor-name
+  assert -> `'datad'`; `install_brokerd_search` ->
+  `install_datad_search`; comment sweep
+- `piker/brokers/_daemon.py` —
+  `_setup_persistent_brokerd()` slimmed (feed-bus
+  alloc moved to the datad fixture)
+- `piker/ui/_app.py`, `piker/ui/cli.py`,
+  `piker/brokers/core.py`, `piker/brokers/cli.py`,
+  `piker/ui/kivy/option_chain.py` — spawn-site
+  repoints
+- `tests/test_services.py` — datad assertions + the
+  no-`brokerd.kraken` negative check
+
+## Human edits
+
+None — committed as generated.

ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.raw.md

@@ -0,0 +1,54 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:12:59Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); backfilled from the live dev
+session transcript per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/data/feed.py`
+
+Generated: `open_feed()` spawn cutover with the
+`maybe_spawn_datad` import done "relative-direct" from
+`._daemon` — NOT via `piker.service` — to dodge a
+partial-init cycle: when `piker.data.feed` loads as
+part of `piker.service.__init__` executing its own
+`piker.data._daemon` import, the service pkg is
+mid-init and its `maybe_spawn_datad` binding does not
+exist yet. The `open_feed_bus()` local-state sanity
+assert flips `'brokerd' in servicename` ->
+`'datad' in servicename` (the only actor-name assert
+in the tree, verified by grep).
+
+> `git log -1 -p --follow -- piker/brokers/_daemon.py`
+
+Generated: `_setup_persistent_brokerd()` slimmed to
+console-log setup + pinned-open ctx; drops the bus
+alloc, the `assert not feed._bus`, the service nursery
+(backend `open_trade_dialog()` ctxs own their task
+trees) and the `eg.ExceptionGroup` handler. The
+`piker ledger` ad-hoc actor enters this same slimmed
+fixture — exactly what it needs.
+
+> `git log -1 -p --follow -- piker/ui/_app.py`
+> `git log -1 -p --follow -- piker/ui/cli.py`
+> `git log -1 -p --follow -- piker/brokers/core.py`
+> `git log -1 -p --follow -- piker/brokers/cli.py`
+> `git log -1 -p --follow -- piker/ui/kivy/option_chain.py`
+> `git log -1 -p --follow -- tests/test_services.py`
+
+Verification (from session): per-suite gates green at
+this commit (services 5-passed incl. the new negative
+assert, feeds 3-passed); a headless live smoke
+(`maybe_open_pikerd` + `open_feed(['xbtusdt.kraken'])`
+on an alt registry port) confirmed quotes flowing via
+`datad.kraken` + `samplerd` with `check_for_service(
+'brokerd.kraken') is None`. Known pre-existing flake
+documented: `test_multi_fill_positions`' second
+in-proc runtime boot wedges ~50% (zombie subactor w/
+unread parent-IPC bytes); reproduced with the split
+fully reverted so NOT a regression of this work.

ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md

@@ -0,0 +1,43 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:13:44Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T171344Z_eee19de0_prompt_io.raw.md
+---
+
+## Prompt
+
+Same session-initiating `brokerd`-split instruction (see
+`20260610T170859Z_75cefe10_prompt_io.md`); this is the
+approved plan's final "stage 4": caps-sec slimming of
+the (live, credentialed) trading actor + the `ib`
+dual-daemon `client_id` collision mitigation flagged in
+the plan's risk register.
+
+## Response summary
+
+`brokerd` loses ALL `piker.data.*` (feed) RPC mods;
+spawn fails fast for datad-only backends with a "use
+paper-mode" error; `ib`'s default api-gw `client_id`
+gets a per-daemon-kind offset so `datad.ib` +
+`brokerd.ib` don't collide on connect.
+
+## Files changed
+
+- `piker/brokers/_daemon.py` — `_data_mods` -> minimal
+  `_brokerd_service_mods`; `broker_init()` reads
+  `_brokerd_mods` (fallback `__enable_modules__`);
+  `spawn_brokerd()` fail-fast via `validate.get_eps()`
+- `piker/brokers/ib/api.py` — role-based `client_id`
+  offset in `load_aio_clients()`
+- `piker/cli/__init__.py` — resolved "expose datad"
+  TODO
+
+## Human edits
+
+None — committed as generated.

ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.raw.md

@@ -0,0 +1,41 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:13:44Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); backfilled from the live dev
+session transcript per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/brokers/_daemon.py`
+
+Generated: the fail-fast originally landed in
+`broker_init()` but was relocated to `spawn_brokerd()`
+mid-implementation after realizing `piker ledger`
+calls `broker_init()` directly even for paper accounts
+on datad-only backends (would have crashed the cli);
+the service-spawn path is the correct enforcement
+seam. Error text:
+
+  Backend 'kucoin' offers NO `brokerd` (live
+  order-control) eps!? It is likely a datad-only
+  provider, use paper-mode for clearing instead.
+
+(verified live via a `trio.run()` unit check.)
+
+> `git log -1 -p --follow -- piker/brokers/ib/api.py`
+
+Generated: in `load_aio_clients()`, when `client_id`
+is the 6116 default: `datad`-named actors offset +16
+(disjoint from `brokerd`'s linear `client_id + i`
+retry range), other non-`brokerd` (ad-hoc test/cli)
+actors +32. Rationale from the plan's risk register:
+post-split BOTH per-broker daemons connect to the same
+TWS/gw endpoint; a shared default id collides and
+burns up to `connect_timeout * retries` (90s) in
+retry cycles.
+
+> `git log -1 -p --follow -- piker/cli/__init__.py`

ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md

@@ -0,0 +1,66 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T17:33:09Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
+scope: docs
+substantive: true
+raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md
+---
+
+## Prompt
+
+The session-initiating instruction, verbatim (this doc
+IS its "mega detailed plan" deliverable):
+
+> ok i want you to become the distributed runtime and
+> concurrency expert for this project - namely acquire
+> a deep understanding of tractor and how it's used.
+> then i want you to atttempt to factor our current
+> brokerd service daemon into 2 daemons:
+>
+> - a brokerd which only hosts live/paper trading
+>   endoint tasks as defined namely within all
+>   piker/brokers/<backend>/broker.py mods
+> - a new `datad` subdaemon which in a separate
+>   subactor serves all the data feed service tasks
+>   namely delivered by endpoints in
+>   piker/brokers/<backend>/feed.py (or similar for
+>   less mature backends) and as more rigorously
+>   defined by the validation machinery in
+>   piker.data.validate.
+>
+> give me a mega detailed plan on how to approach
+> this, and a staged approach for the implementation.
+
+Plan-shaping user decisions (in-session Q&A): sibling
+topology under `pikerd`; hard cutover staged by layer;
+trading-only EMS-lazy-spawned `brokerd`. The human
+staged the doc copy into `ai/claude-code/plans/` and
+requested this commit after the 7 implementation
+commits landed.
+
+## Response summary
+
+The staged design doc for the `brokerd` -> (`datad` +
+`brokerd`) split: context, target supervision topology,
+load-bearing verified facts, per-stage file-level
+changes with gates, risk register and an end-to-end
+verification matrix. Produced via 3 parallel explore
+agents + 1 architect agent + human Q&A before any code
+was written; all 7 implementation commits in this
+branch reference its stages in their provenance
+entries.
+
+## Files changed
+
+- `ai/claude-code/plans/datad_service.md` — the design
+  plan doc (human-staged copy of the AI-authored plan)
+
+## Human edits
+
+None to the doc content — committed as generated. NB:
+the implementation deviated from the plan-as-written in
+4 places, see the raw file's deviation log.

ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.raw.md

@@ -0,0 +1,51 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T17:33:09Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
+---
+
+NOTE: diff-ref mode entry; the AI output here is the
+committed doc itself (a design plan, not code) so no
+verbatim copy is duplicated:
+
+> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md`
+
+## Implementation deviation log
+
+Where the landed 7-commit series differs from the plan
+as written (recorded for reviewer transparency; the
+plan doc is committed unmodified):
+
+1. Commit ordering: the `pytest` config-dir isolation
+   fix lands BEFORE the `tractor`-API drift port. The
+   plan's stage gates assumed a runnable+isolated test
+   baseline; per-commit gating exposed that without
+   isolation the paper-EMS test reads the user's real
+   (polluted) `account.kraken.paper.toml` and reddens.
+2. The plan's "stage 0: full pytest green" gate
+   required first repairing pre-existing branch
+   breakage vs `tractor` git `main` (boot
+   `AttributeError`, stale discovery/exc/position
+   APIs) — that repair became its own commit
+   ("Port service+tests to latest `tractor` APIs")
+   rather than plan-stage work.
+3. Stage-4 fail-fast placement: the plan said
+   `broker_init()` raises on brokerd-ep-less backends;
+   implementation moved the raise to `spawn_brokerd()`
+   since `piker ledger` calls `broker_init()` directly
+   even for paper accounts on datad-only backends.
+4. `brokers/_daemon.py` change grouping: the
+   import-cleanup hunks (`exceptiongroup`, `_FeedsBus`
+   type-only import) landed with the caps-sec slim
+   commit instead of the fixture-slim commit, keeping
+   each intermediate tree import-clean without
+   sub-hunk surgery.
+
+Also of record: the plan's "Verification" matrix was
+executed as written (per-suite gates each stage, the
+headless datad-feed smoke, the kucoin fail-fast unit
+check); the known pre-existing ~50% second-runtime-boot
+test wedge was characterized and excluded as a
+regression via revert-testing.

ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md

@@ -0,0 +1,60 @@
+---
+model: claude-fable-5[1m]
+service: claude
+session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
+timestamp: 2026-06-10T21:35:49Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
+scope: code
+substantive: true
+raw_file: 20260610T213549Z_f084e899_prompt_io.raw.md
+---
+
+## Prompt
+
+User bug report from live `ib` testing of the
+(datad|brokerd)-split branch:
+
+> ok doing some testing and noticing live orders do not
+> work otb, pikerd show this on live submission,
+> [pdb traceback: `RuntimeError("Can not order
+> {symbol}, no live feed?")` at `ib/api.py:1152`
+> `submit_limit()` with `self._contracts == {}` in the
+> `brokerd.ib` actor + a `TrioTaskExited` teardown
+> cascade]
+
+Follow-up user direction fixing the design intent:
+
+> i think the main thing here is that [brokerd] should
+> be able to always submit orders without a live feed
+> being up and registered in the same subactor right?
+
+## Response summary
+
+Root cause: pre-split the feed code populated the
+actor-local `Client._contracts` cache (via
+`get_mkt_info()` writing `mkt.bs_fqme` keys) in the
+SAME process as order submission; post-split the
+trading actor's client is never warmed. Fix: lazily
+qualify + cache the contract per order request in
+`handle_order_requests()` by running the same
+`get_mkt_info(fqme, proxy=...)` ep the feed side uses,
+plus per-order error relay (`BrokerdError`) so one bad
+submission can't crash the whole trades dialog (the
+`TrioTaskExited` storm was teardown cascade from the
+original raise).
+
+## Files changed
+
+- `piker/brokers/ib/broker.py` — thread `proxies` into
+  `handle_order_requests()`; lazy contract qualify on
+  cache-miss; guard `submit_limit()` w/ `BrokerdError`
+  relay; uncomment the (anticipatory) `get_mkt_info`
+  import
+- `piker/brokers/ib/api.py` — fix the non-f-string
+  raise msg + document the new qualification contract
+
+## Human edits
+
+None — committed as generated. Live `ib` order retest
+performed by the human post-commit.

ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.raw.md

@@ -0,0 +1,57 @@
+---
+model: claude-fable-5[1m]
+service: claude
+timestamp: 2026-06-10T21:35:49Z
+git_ref: datad_service
+diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
+---
+
+NOTE: diff-ref mode entry (code committed in the same
+commit as this log); recorded from the live debug
+session per the `/prompt-io` skill rules.
+
+> `git log -1 -p --follow -- piker/brokers/ib/broker.py`
+> `git log -1 -p --follow -- piker/brokers/ib/api.py`
+
+Key diagnostic chain (from session):
+
+- pdb showed `Client._contracts == {}` inside
+  `brokerd.ib`'s `submit_limit()`; the cache has
+  exactly TWO write sites: `Client.find_contracts()`
+  (api.py, keys `f'{sym}.{exch}.ib'`) and
+  `symbols.get_mkt_info()` (key `mkt.bs_fqme`, eg.
+  'nvda.nasdaq' — NO `.ib` suffix).
+- `BrokerdOrder.symbol` arrives as the bs_fqme form
+  ('nvda.nasdaq') so ONLY the `get_mkt_info()` write
+  site produces the key `submit_limit()` reads —
+  ie. pre-split it was the feed's in-proc
+  `get_mkt_info(sym, proxy=proxy)` call keeping orders
+  working, NOT `find_contracts()`.
+- the existing TODO at `symbols.py:642-644` literally
+  predicted this: "this is going to be problematic
+  if/when we split out the datad vs. brokerd actors
+  since the mktmap lookup table will now be
+  inaccessible.."
+- instance identity verified: `proxy._aio_ns` IS the
+  same `Client` obj as `_accounts2clients[account]`
+  (both sourced from the `load_aio_clients()` cache via
+  `open_client_proxies()`), so a brokerd-side
+  `get_mkt_info(fqme, proxy=proxies[account])` warms
+  exactly the dict `submit_limit()` reads. It also
+  populates `client._cons2mkts` which the
+  position-audit path (`broker.py` backup-table code)
+  needs in this actor anyway.
+- the `TrioTaskExited` storm in the user's log
+  (`recv_trade_updates`, `open_aio_client_method_relay`
+  aio tasks) is teardown cascade: the raise crashed
+  `handle_order_requests` -> nursery teardown ripped
+  the trio sides of still-running aio relay tasks.
+  Hence the added per-order try/except ->
+  `BrokerdError` relay hardening so a single bad
+  submission degrades to an EMS error msg instead of
+  killing the backend's entire order-ctl dialog.
+
+Verification: `tests/test_services.py` (5 passed) +
+`tests/test_ems.py` (6 passed) regression-green; live
+`ib` submission retest delegated to the human (needs a
+running TWS/gw).

ai/prompt-io/claude/README.md

@@ -0,0 +1,34 @@
+# AI Prompt I/O Log — claude
+
+This directory tracks prompt inputs and model
+outputs for AI-assisted development using
+`claude` (claude-code CLI).
+
+## Policy
+
+Prompt logging follows the
+[NLNet generative AI policy][nlnet-ai].
+All substantive AI contributions are logged
+with:
+- Model name and version
+- Timestamps
+- The prompts that produced the output
+- Unedited model output (`.raw.md` files)
+
+[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
+
+## Usage
+
+Entries are created by the `/prompt-io` skill
+or automatically via `/commit-msg` integration.
+
+Each commit carrying AI-generated changes links
+to its provenance entry via a `Prompt-IO:`
+commit-msg trailer; entries use "diff-ref mode"
+(pointers into `git log -p` instead of verbatim
+code copies) to avoid duplicating committed
+code.
+
+Human contributors remain accountable for all
+code decisions. AI-generated content is never
+presented as human-authored work.

ai/prompt-io/opencode/20260721T205228Z_58ffe487_prompt_io.md

@@ -0,0 +1,64 @@
+---
+model: gpt-5.6-sol
+provider: openai
+service: opencode
+session: ses_0799212ebffe42arY96czXn89F
+timestamp: 2026-07-21T20:52:28Z
+git_ref: 58ffe487
+scope: config
+substantive: true
+raw_file: 20260721T205228Z_58ffe487_prompt_io.raw.md
+---
+
+## Prompt
+
+The user asked to scan Claude's repo-local skills and
+documentation, summarize their specializations, and then
+rework repo-specific skills for use across coding
+harnesses, particularly OpenCode. The user intends to
+choose one of Claude's unfinished tasks afterward.
+
+## Response summary
+
+Standardized the tracked piker skills on portable Agent
+Skills frontmatter, made `commit-msg` harness-aware,
+documented OpenCode discovery and command delegation, and
+fixed an unsafe `np.searchsorted()` example found during
+the audit. Preserved the existing `.claude` artifact path
+as a shared legacy location to avoid breaking established
+commit workflows.
+
+## Files changed
+
+- `.claude/skills/commit-msg/SKILL.md` - portable,
+  harness-aware commit-message workflow
+- `.claude/skills/commit-msg/style-guide-reference.md` -
+  generic coding-harness attribution
+- `.claude/skills/piker-conc-expert/SKILL.md` - portable
+  frontmatter
+- `.claude/skills/piker-profiling/SKILL.md` - portable
+  frontmatter
+- `.claude/skills/piker-slang/SKILL.md` - portable
+  frontmatter
+- `.claude/skills/pyqtgraph-optimization/SKILL.md` -
+  portable frontmatter
+- `.claude/skills/timeseries-optimization/SKILL.md` -
+  portable frontmatter and safe search bounds handling
+- `ai/README.md` - shared-skill integration model
+- `ai/claude-code/README.md` - cross-harness Claude Code
+  documentation
+- `ai/opencode/README.md` - OpenCode usage and discovery
+  documentation
+- `.opencode/commands/commit-msg.md` - thin OpenCode
+  command delegating to the shared skill
+- `ai/prompt-io/opencode/README.md` - OpenCode provenance
+  policy
+- `ai/prompt-io/opencode/20260721T205228Z_58ffe487_prompt_io.raw.md`
+  - unedited response record
+- `ai/prompt-io/opencode/20260721T205228Z_58ffe487_prompt_io.md`
+  - provenance metadata and summary
+
+## Human edits
+
+None - generated changes have not yet been reviewed or
+edited by the human.

ai/prompt-io/opencode/20260721T205228Z_58ffe487_prompt_io.raw.md

@@ -0,0 +1,33 @@
+---
+model: gpt-5.6-sol
+provider: openai
+service: opencode
+timestamp: 2026-07-21T20:52:28Z
+git_ref: 58ffe487
+diff_cmd: git diff HEAD~1..HEAD
+---
+
+The repo-local skill inventory contains seven domains:
+commit-message generation, actor-runtime concurrency,
+distributed profiling, project communication style,
+Python source style, PyQtGraph rendering, and NumPy/Polars
+timeseries optimization.
+
+I converted the six tracked repo-specific skills to the
+portable Agent Skills frontmatter subset so Claude Code
+and OpenCode can load the same source files. The
+commit-message workflow now gathers live git context at
+execution time, uses runtime harness/model attribution,
+preserves human-owned task state, and keeps the existing
+message artifact paths for compatibility.
+
+I added OpenCode integration documentation and updated
+the shared and Claude Code docs to describe a single
+cross-harness skill source with thin harness commands.
+During validation I also corrected an out-of-bounds
+`np.searchsorted()` example in the timeseries skill.
+
+The unfinished artifacts found during the scan are a
+shared-memory migration plan and a gap-annotation
+performance handoff. The earlier datad split plan appears
+implemented and is now captured by the concurrency skill.

ai/prompt-io/opencode/README.md

@@ -0,0 +1,20 @@
+# AI Prompt I/O Log - OpenCode
+
+This directory tracks prompt inputs and model outputs for
+AI-assisted development using OpenCode.
+
+## Policy
+
+Prompt logging follows the
+[NLNet generative AI policy][nlnet-ai]. Substantive AI
+contributions are logged with the model, timestamp,
+prompt, response, and affected files.
+
+[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
+
+## Usage
+
+Entries are created by the `prompt-io` skill. Human
+contributors remain accountable for all code decisions;
+AI-generated content is not presented as human-authored
+work.

config/brokers.toml

@@ -32,7 +32,14 @@ option.log.disabled = true
 
 
 [kraken]
-key_descr = ''
+# the reference fiat asset as can be set
+# in an account's web-trading-UI prefs.
+src_fiat = 'usd'
+
+# NOTE for account defs, the following
+# lines must match as follows.
+accounts.spot = 'spot'
+key_descr = 'spot'
 api_key = ''
 secret = ''
 # ------ kraken ------

config/conf.toml

@@ -1,8 +1,26 @@
 [network]
 pikerd = [
-  '/ipv4/127.0.0.1/tcp/6116',  # std localhost daemon-actor tree
-  # '/uds/6116',  # TODO std uds socket file
+  # TCP localhost loopback
+  '/ip4/127.0.0.1/tcp/6116',
+
+  # same but UDS
+  '/unix/run/user/1000/piker/pikerd.sock',
 ]
+chart = [
+  '/ip4/127.0.0.1/tcp/3003',
+  '/unix/run/user/1000/piker/chart.sock',
+]
+# the service-actor registry endpoint;
+# other pikerd trees contact this to discover
+# actors.
+# XXX if absent, pikerd binds the registry
+# on its own tpt_bind_addrs.
+# regd = ['/ip4/127.0.0.1/tcp/6116']
+
+# chart = [
+#   '/ip4/127.0.0.1/tcp/3333',
+#   '/unix/run/user/1000/piker/chart@3333.sock',
+# ]
 
 
 [ui]

flake.lock

@@ -2,11 +2,11 @@
   "nodes": {
     "nixpkgs": {
       "locked": {
-        "lastModified": 1765779637,
-        "narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
+        "lastModified": 1784497964,
+        "narHash": "sha256-vlHUuqAcbcH2RKmHbPiuQzbv1pnzzavXnI62RD0bqCU=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
+        "rev": "241313f4e8e508cb9b13278c2b0fa25b9ca27163",
         "type": "github"
       },
       "original": {

flake.nix

@@ -22,12 +22,14 @@
 
           # do store-path extractions
           qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
+          zlibStorePath = lib.getLib pkgs.zlib;
           # ?TODO? can remove below since manual linking not needed?
           # qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
 
           # XXX NOTE XXX, for now we overlay specific pkgs via
           # a major-version-pinned-`cpython`
           cpython = "python313";
+          python = pkgs.${cpython};
           pypkgs = pkgs."${cpython}Packages";
         in
         {
@@ -46,7 +48,7 @@
               qt6.qtbase
 
               uv
-              python313  # ?TODO^ how to set from `cpython` above?
+              python
               pypkgs.pyqt6
               pypkgs.pyqt6-sip
               pypkgs.qtpy
@@ -70,6 +72,7 @@
 
               # link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
               LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
+              LD_LIBRARY_PATH="${zlibStorePath}/lib:$LD_LIBRARY_PATH"
 
               export LD_LIBRARY_PATH
 
@@ -86,15 +89,31 @@
               export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
 
               # ------ uv ------
+              # ignore any venv inherited from an outer checkout
+              unset VIRTUAL_ENV
               # - always use the ./py313/ venv-subdir
               export UV_PROJECT_ENVIRONMENT="py313"
+
+              # Rebuild after a flake update changes the Nix Python closure.
+              venv_python="$UV_PROJECT_ENVIRONMENT/bin/python"
+              nix_python="$(readlink -f "${python}/bin/python3")"
+              if test -e "$venv_python" \
+                && test "$(readlink -f "$venv_python")" != "$nix_python"
+              then
+                uv venv \
+                  --clear \
+                  --python "$nix_python" \
+                  "$UV_PROJECT_ENVIRONMENT"
+              fi
+
               # sync project-env with all extras
               uv sync --dev --all-extras --no-group lint
+              source "$UV_PROJECT_ENVIRONMENT/bin/activate"
 
               # ------ TIPS ------
               # NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
               # run the `nix develop` cmd with,
-              # >> nix develop -c uv run xonsh
+              # >> nix develop -c xonsh
             '';
           };
         }

piker/accounting/_ledger.py

@@ -324,10 +324,13 @@ def load_ledger(
     ldir: Path = (
         dirpath
         or
-        config._config_dir / 'accounting' / 'ledgers'
+        config.get_conf_dir() / 'accounting' / 'ledgers'
     )
     if not ldir.is_dir():
-        ldir.mkdir()
+        ldir.mkdir(
+            parents=True,
+            exist_ok=True,
+        )
 
     fname = f'trades_{brokername}_{acctid}.toml'
     fpath: Path = ldir / fname

piker/accounting/_pos.py

@@ -785,9 +785,16 @@ def load_account(
     legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
     fn: str = f'account.{brokername}.{acctid}.toml'
 
-    dirpath: Path = dirpath or (config._config_dir / 'accounting')
+    dirpath: Path = (
+        dirpath
+        or
+        (config.get_conf_dir() / 'accounting')
+    )
     if not dirpath.is_dir():
-        dirpath.mkdir()
+        dirpath.mkdir(
+            parents=True,
+            exist_ok=True,
+        )
 
     conf, path = config.load(
         path=dirpath / fn,

piker/accounting/cli.py

@@ -143,12 +143,15 @@ def sync(
                     # (what the EMS normally does internall) B)
                     open_brokerd_dialog(
                         brokermod,
-                        portal,
                         exec_mode=(
                             'paper'
                             if account == 'paper'
                             else 'live'
                         ),
+                        # use our own ad-hoc-spawned actor,
+                        # do NOT (spawn and) use the
+                        # `brokerd.<broker>` service daemon!
+                        portal=portal,
                         loglevel=loglevel,
                     ) as (
                         brokerd_stream,

piker/brokers/__init__.py

@@ -31,6 +31,7 @@ from piker.log import (
 from ._util import (
     BrokerError,
     SymbolNotFound,
+    MarketNotFound as MarketNotFound,
     NoData,
     DataUnavailable,
     DataThrottle,

piker/brokers/_daemon.py

@@ -25,10 +25,8 @@ from contextlib import (
 )
 from types import ModuleType
 from typing import (
-    TYPE_CHECKING,
     AsyncContextManager,
 )
-import exceptiongroup as eg
 
 import tractor
 import trio
@@ -40,27 +38,20 @@ from piker.log import (
 from . import _util
 from . import get_brokermod
 
-if TYPE_CHECKING:
-    from ..data import _FeedsBus
-
 log = get_logger(name=__name__)
 
-# `brokerd` enabled modules
-# TODO: move this def to the `.data` subpkg..
+# `brokerd`-actor-always-enabled mods.
 # NOTE: keeping this list as small as possible is part of our caps-sec
-# model and should be treated with utmost care!
-_data_mods: str = [
-    'piker.brokers.core',
-    'piker.brokers.data',
+# model and should be treated with utmost care! In particular NO
+# `piker.data.*` feed mods should be enabled in this (live,
+# credentialed) trading actor; all data-feed serving is the
+# domain of the `datad.<broker>` sibling daemon, see
+# `piker.data._daemon._datad_service_mods`.
+_brokerd_service_mods: list[str] = [
     'piker.brokers._daemon',
-    'piker.data',
-    'piker.data.feed',
-    'piker.data._sampling'
 ]
 
 
-# TODO: we should rename the daemon to datad prolly once we split up
-# broker vs. data tasks into separate actors?
 @tractor.context
 async def _setup_persistent_brokerd(
     ctx: tractor.Context,
@@ -69,9 +60,15 @@ async def _setup_persistent_brokerd(
 
 ) -> None:
     '''
-    Allocate a actor-wide service nursery in ``brokerd``
-    such that feeds can be run in the background persistently by
-    the broker backend as needed.
+    Trading-only daemon (lifetime) fixture: console logging
+    setup and a pinned-open context for service mgmt.
+
+    All data-feed-bus state now lives in the (data-feed-only)
+    `datad.<brokername>` sibling daemon, see
+    `piker.data._daemon._setup_persistent_datad()`; this
+    actor hosts only the backend's `open_trade_dialog()`
+    (live order-control) ep-task(s) which manage their own
+    task trees per `tractor.Context`.
 
     '''
     # NOTE: we only need to setup logging once (and only) here
@@ -87,47 +84,13 @@ async def _setup_persistent_brokerd(
     )
     assert log.name == _util.subsys
 
-    # further, set the log level on any broker broker specific
-    # logger instance.
-
-    from piker.data import feed
-    assert not feed._bus
-
-    # allocate a nursery to the bus for spawning background
-    # tasks to service client IPC requests, normally
-    # `tractor.Context` connections to explicitly required
-    # `brokerd` endpoints such as:
-    # - `stream_quotes()`,
-    # - `manage_history()`,
-    # - `allocate_persistent_feed()`,
-    # - `open_symbol_search()`
-    # NOTE: see ep invocation details inside `.data.feed`.
-    try:
-        async with (
-            # tractor.trionics.collapse_eg(),
-            trio.open_nursery() as service_nursery
-        ):
-            bus: _FeedsBus = feed.get_feed_bus(
-                brokername,
-                service_nursery,
-            )
-            assert bus is feed._bus
-
     # unblock caller
     await ctx.started()
 
-            # we pin this task to keep the feeds manager active until the
+    # we pin this task to keep the daemon active until the
     # parent actor decides to tear it down
     await trio.sleep_forever()
 
-    except eg.ExceptionGroup:
-        # TODO: likely some underlying `brokerd` IPC connection
-        # broke so here we handle a respawn and re-connect attempt!
-        # This likely should pair with development of the OCO task
-        # nusery in dev over @ `tractor` B)
-        # https://github.com/goodboy/tractor/pull/363
-        raise
-
 
 def broker_init(
     brokername: str,
@@ -147,8 +110,10 @@ def broker_init(
 
     This includes:
     - load the appropriate <brokername>.py pkg module,
-    - reads any declared `__enable_modules__: listr[str]` which will be
-      passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
+    - reads any declared `_brokerd_mods: list[str]` (falling
+      back to the full `__enable_modules__` set for
+      not-yet-split backends) which will be passed to
+      `tractor.ActorNursery.start_actor(enable_modules=)`
       at actor start time,
     - deliver a references to the daemon lifetime fixture, which
       for now is always the `_setup_persistent_brokerd()` context defined
@@ -182,9 +147,15 @@ def broker_init(
         modpath,
     ]
     for submodname in getattr(
+        brokermod,
+        '_brokerd_mods',
+        # fallback for (flat, less mature) backends which
+        # don't yet declare a daemon-kind mod split.
+        getattr(
             brokermod,
             '__enable_modules__',
             [],
+        ),
     ):
         subpath: str = f'{modpath}.{submodname}'
         enabled.append(subpath)
@@ -212,6 +183,22 @@ async def spawn_brokerd(
         f'backend: {brokername!r}'
     )
 
+    # fail fast on (data-only) backends which don't offer
+    # ANY live order-control eps; the caller should instead
+    # be using paper-mode (and thus never spawning us)!
+    from ..data.validate import get_eps
+    brokerd_eps: dict = get_eps(
+        get_brokermod(brokername),
+        'brokerd',
+    )
+    if not brokerd_eps:
+        raise RuntimeError(
+            f'Backend {brokername!r} offers NO `brokerd` '
+            f'(live order-control) eps!?\n'
+            f'It is likely a datad-only provider, use '
+            f'paper-mode for clearing instead.\n'
+        )
+
     (
         brokermode,
         tractor_kwargs,
@@ -233,7 +220,11 @@ async def spawn_brokerd(
     dname: str = tractor_kwargs.pop('name')  # f'brokerd.{brokername}'
     portal = await Services.actor_n.start_actor(
         dname,
-        enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
+        enable_modules=list(dict.fromkeys(
+            _brokerd_service_mods
+            +
+            tractor_kwargs.pop('enable_modules')
+        )),
         debug_mode=Services.debug_mode,
         **tractor_kwargs
     )

piker/brokers/_util.py

@@ -20,10 +20,17 @@ Handy cross-broker utils.
 """
 from __future__ import annotations
 # from functools import partial
+from typing import (
+    Type,
+)
 
 import json
 import httpx
 import logging
+from msgspec import Struct
+from tractor._exceptions import (
+    reg_err_types,
+)
 
 from piker.log import (
     colorize_json,
@@ -59,6 +66,10 @@ class SymbolNotFound(BrokerError):
     "Symbol not found by broker search"
 
 
+class MarketNotFound(SymbolNotFound):
+    "Mkt-pair not found by broker search"
+
+
 # TODO: these should probably be moved to `.tsp/.data`?
 class NoData(BrokerError):
     '''
@@ -97,6 +108,19 @@ class DataThrottle(BrokerError):
     '''
     # TODO: add in throttle metrics/feedback
 
+class SchemaMismatch(BrokerError):
+    '''
+    Market `Pair` fields mismatch, likely due to provider API update.
+
+    '''
+
+# auto-register all `BrokerError` subtypes for
+# tractor IPC exc-marshalling.
+reg_err_types([
+    BrokerError,
+    *BrokerError.__subclasses__(),
+])
+
 
 def resproc(
     resp: httpx.Response,
@@ -123,3 +147,45 @@ def resproc(
         log.debug(f"Received json contents:\n{colorize_json(msg)}")
 
     return msg if return_json else resp
+
+
+def get_or_raise_on_pair_schema_mismatch(
+    pair_type: Type[Struct],
+    fields_data: dict,
+    provider_name: str,
+    api_url: str|None = None,
+) -> Struct:
+    '''
+    Boilerplate helper around assset-`Pair` field schema mismatches,
+    normally due to provider API updates.
+
+    '''
+    try:
+        pair: Struct = pair_type(**fields_data)
+        return pair
+    except TypeError as err:
+
+        from tractor.devx.pformat import ppfmt
+        repr_data: str = ppfmt(fields_data)
+        report: str = (
+            f'Field mismatch we need to codify!\n'
+            f'\n'
+            f'{pair_type!r}({repr_data})'
+            f'\n'
+            f'^^^ {err.args[0]!r} ^^^\n'
+            f'\n'
+            f"Don't panic, prolly {provider_name!r} "
+            f"changed their symbology schema..\n"
+        )
+        if (
+            api_url
+            or
+            (api_url := pair_type._api_url)
+        ):
+            report += (
+                f'\n'
+                f'Check out their API docs here:\n'
+                f'{api_url}\n'
+            )
+
+        raise SchemaMismatch(report) from err

piker/brokers/binance/__init__.py

@@ -52,9 +52,25 @@ __all__ = [
 ]
 
 
-# `brokerd` modules
-__enable_modules__: list[str] = [
+# per-daemon-kind (sub)mod groups: declares which of our
+# submods host the eps run by each daemon-actor kind as
+# defined by `piker.data.validate._eps`.
+# NOTE: `get_mkt_info` and `open_symbol_search` both live
+# in `.feed` for this backend (no `symbols.py`).
+_brokerd_mods: list[str] = [
     'api',
-    'feed',
     'broker',
 ]
+
+_datad_mods: list[str] = [
+    'api',
+    'feed',
+]
+
+
+# tractor RPC enable arg
+__enable_modules__: list[str] = list(dict.fromkeys(
+    _brokerd_mods
+    +
+    _datad_mods
+))

piker/brokers/binance/api.py

@@ -49,6 +49,9 @@ from piker import config
 from piker.clearing._messages import (
     Order,
 )
+from piker.brokers._util import (
+    get_or_raise_on_pair_schema_mismatch,
+)
 from piker.accounting import (
     Asset,
     digits_to_dec,
@@ -370,20 +373,12 @@ class Client:
                 item['filters'] = filters
 
             pair_type: Type = PAIRTYPES[venue]
-            try:
-                pair: Pair = pair_type(**item)
-            except Exception as e:
-                e.add_note(
-                    f'\n'
-                    f'New or removed field we need to codify!\n'
-                    f'pair-type: {pair_type!r}\n'
-                    f'\n'
-                    f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
-                    f'Check out their API docs here:\n'
-                    f'\n'
-                    f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
+            pair: Pair = get_or_raise_on_pair_schema_mismatch(
+                pair_type=pair_type,
+                fields_data=item,
+                provider_name='binance',
+                api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
             )
-                raise
             pair_table[pair.symbol.upper()] = pair
 
             # update an additional top-level-cross-venue-table
@@ -581,8 +576,8 @@ class Client:
         self,
         mkt: MktPair,
 
-        start_dt: datetime | None = None,
-        end_dt: datetime | None = None,
+        start_dt: datetime|None = None,
+        end_dt: datetime|None = None,
 
         as_np: bool = True,
 
@@ -609,7 +604,11 @@ class Client:
         start_time = binance_timestamp(start_dt)
         end_time = binance_timestamp(end_dt)
 
-        bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
+        import tractor
+        with tractor.devx.maybe_open_crash_handler():
+            bs_pair: Pair = self._pairs[
+                mkt.bs_fqme.upper()
+            ]
 
         # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
         bars = await self._api(

piker/brokers/binance/feed.py

@@ -48,6 +48,7 @@ import tractor
 from piker.brokers import (
     open_cached_client,
     NoData,
+    MarketNotFound,
 )
 from piker._cacheables import (
     async_lifo_cache,
@@ -203,9 +204,13 @@ async def stream_messages(
                 yield 'trade', piker_quote
 
 
-def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
+def make_sub(
+    pairs: list[str],
+    sub_name: str,
+    uid: int,
+) -> dict[str, str]:
     '''
-    Create a request subscription packet dict.
+    Create a request subscription packet `dict`.
 
     - spot:
       https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
@@ -301,6 +306,10 @@ async def get_mkt_info(
 
     # uppercase since kraken bs_mktid is always upper
     if 'binance' not in fqme.lower():
+        log.warning(
+            f'Missing `.<provider>` part in fqme ??\n'
+            f'fqme: {fqme!r}\n'
+        )
         fqme += '.binance'
 
     mkt_mode: str = ''
@@ -315,6 +324,24 @@ async def get_mkt_info(
     venue: str = venue.upper()
     venue_lower: str = venue.lower()
 
+    if not venue:
+        if expiry:
+            expiry = f'.{expiry}'
+        expected: str = (
+            f'{mkt_ep}'
+            f'.<venue>'
+            f'{expiry}'
+            f'.{broker}'
+        )
+        raise MarketNotFound(
+            f'Invalid or missing .<venue> part in fqme?\n'
+            f'\n'
+            f'fqme: {fqme!r}\n'
+            f'expected-form>> {expected}\n'
+            f'\n'
+            f'Maybe you are missing a ".spot." ?\n'
+        )
+
     # XXX TODO: we should change the usdtm_futes name to just
     # usdm_futes (dropping the tether part) since it turns out that
     # there are indeed USD-tokens OTHER THEN tether being used as
@@ -332,7 +359,8 @@ async def get_mkt_info(
         # TODO: handle coinm futes which have a margin asset that
         # is some crypto token!
         # https://binance-docs.github.io/apidocs/delivery/en/#exchange-information
-        or 'btc' in venue_lower
+        or
+        'btc' in venue_lower
     ):
         return None
 
@@ -343,16 +371,21 @@ async def get_mkt_info(
 
     if (
         venue
-        and 'spot' not in venue_lower
+        and
+        'spot' not in venue_lower
 
         # XXX: catch all in case user doesn't know which
         # venue they want (usdtm vs. coinm) and we can choose
         # a default (via config?) once we support coin-m APIs.
-        or 'perp' in venue_lower
+        or
+        'perp' in venue_lower
     ):
         if not mkt_mode:
             mkt_mode: str = f'{venue_lower}_futes'
 
+    # tracing
+    # await tractor.pause()
+
     async with open_cached_client(
         'binance',
     ) as client:

piker/brokers/binance/venues.py

@@ -20,6 +20,7 @@ Per market data-type definitions and schemas types.
 """
 from __future__ import annotations
 from typing import (
+    ClassVar,
     Literal,
 )
 from decimal import Decimal
@@ -203,6 +204,8 @@ class FutesPair(Pair):
     # NOTE: see `.data._symcache.SymbologyCache.load()` for why
     ns_path: str = 'piker.brokers.binance:FutesPair'
 
+    _api_url: ClassVar[str] = 'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
+
     # NOTE: for compat with spot pairs and `MktPair.src: Asset`
     # processing..
     @property

piker/brokers/cli.py

@@ -35,7 +35,7 @@ from piker.log import (
     get_logger,
 )
 from ..service import (
-    maybe_spawn_brokerd,
+    maybe_spawn_datad,
     maybe_open_pikerd,
 )
 from ..brokers import (
@@ -187,7 +187,7 @@ def brokercheck(config, broker):
 
     '''
     async def bcheck_main():
-        async with maybe_spawn_brokerd(broker) as portal:
+        async with maybe_spawn_datad(broker) as portal:
             await portal.run(run_test, broker)
             await portal.cancel_actor()
 
@@ -317,7 +317,7 @@ def record(config, rate, name, dhost, filename):
         return
 
     async def main(tries):
-        async with maybe_spawn_brokerd(
+        async with maybe_spawn_datad(
             tries=tries, loglevel=loglevel
         ) as portal:
             # run app "main"

piker/brokers/core.py

@@ -30,7 +30,7 @@ import trio
 
 from piker.log import get_logger
 from . import get_brokermod
-from ..service import maybe_spawn_brokerd
+from ..service import maybe_spawn_datad
 from . import open_cached_client
 from ..accounting import MktPair
 
@@ -172,7 +172,7 @@ async def symbol_search(
         # await tractor.devx._debug.maybe_init_greenback()
         # tractor.pause_from_sync()
 
-        async with maybe_spawn_brokerd(
+        async with maybe_spawn_datad(
             mod.name,
             infect_asyncio=getattr(
                 mod,

piker/brokers/data.py

@@ -425,7 +425,7 @@ class DataFeed:
 async def stream_to_file(
     watchlist_name: str,
     filename: str,
-    portal: tractor._portal.Portal,
+    portal: tractor.Portal,
     tickers: List[str],
     brokermod: ModuleType,
     rate: int,

piker/brokers/deribit/__init__.py

@@ -47,13 +47,25 @@ __all__ = [
 ]
 
 
-# tractor RPC enable arg
-__enable_modules__: list[str] = [
+# per-daemon-kind (sub)mod groups: declares which of our
+# submods host the eps run by each daemon-actor kind as
+# defined by `piker.data.validate._eps`.
+# NOTE: datad-only backend (no `broker.py` yet)!
+_brokerd_mods: list[str] = []
+
+_datad_mods: list[str] = [
     'api',
     'feed',
-#   'broker',
 ]
 
+
+# tractor RPC enable arg
+__enable_modules__: list[str] = list(dict.fromkeys(
+    _brokerd_mods
+    +
+    _datad_mods
+))
+
 # passed to ``tractor.ActorNursery.start_actor()``
 _spawn_kwargs = {
     'infect_asyncio': True,

piker/brokers/deribit/api.py

@@ -23,7 +23,6 @@ from contextlib import (
     asynccontextmanager as acm,
 )
 from datetime import datetime
-from functools import partial
 import time
 from typing import (
     Any,
@@ -38,7 +37,7 @@ from rapidfuzz import process as fuzzy
 import numpy as np
 from tractor.trionics import (
     broadcast_receiver,
-    maybe_open_context
+    maybe_open_context,
     collapse_eg,
 )
 from tractor import to_asyncio
@@ -524,13 +523,12 @@ async def maybe_open_feed_handler() -> trio.abc.ReceiveStream:
 
 
 async def aio_price_feed_relay(
+    chan: to_asyncio.LinkedTaskChannel,
     fh: FeedHandler,
     instrument: Symbol,
-    from_trio: asyncio.Queue,
-    to_trio: trio.abc.SendChannel,
 ) -> None:
     async def _trade(data: dict, receipt_timestamp):
-        to_trio.send_nowait(('trade', {
+        chan.send_nowait(('trade', {
             'symbol': cb_sym_to_deribit_inst(
                 str_to_cb_sym(data.symbol)).lower(),
             'last': data,
@@ -540,7 +538,7 @@ async def aio_price_feed_relay(
         }))
 
     async def _l1(data: dict, receipt_timestamp):
-        to_trio.send_nowait(('l1', {
+        chan.send_nowait(('l1', {
             'symbol': cb_sym_to_deribit_inst(
                 str_to_cb_sym(data.symbol)).lower(),
             'ticks': [
@@ -570,7 +568,7 @@ async def aio_price_feed_relay(
             install_signal_handlers=False)
 
     # sync with trio
-    to_trio.send_nowait(None)
+    chan.started_nowait(None)
 
     await asyncio.sleep(float('inf'))
 
@@ -581,11 +579,9 @@ async def open_price_feed(
 ) -> trio.abc.ReceiveStream:
     async with maybe_open_feed_handler() as fh:
         async with to_asyncio.open_channel_from(
-            partial(
             aio_price_feed_relay,
-                fh,
-                instrument
-            )
+            fh=fh,
+            instrument=instrument,
         ) as (chan, first):
             yield chan
 
@@ -611,10 +607,9 @@ async def maybe_open_price_feed(
 
 
 async def aio_order_feed_relay(
+    chan: to_asyncio.LinkedTaskChannel,
     fh: FeedHandler,
     instrument: Symbol,
-    from_trio: asyncio.Queue,
-    to_trio: trio.abc.SendChannel,
 ) -> None:
     async def _fill(data: dict, receipt_timestamp):
         breakpoint()
@@ -637,7 +632,7 @@ async def aio_order_feed_relay(
             install_signal_handlers=False)
 
     # sync with trio
-    to_trio.send_nowait(None)
+    chan.started_nowait(None)
 
     await asyncio.sleep(float('inf'))
 
@@ -648,11 +643,9 @@ async def open_order_feed(
 ) -> trio.abc.ReceiveStream:
     async with maybe_open_feed_handler() as fh:
         async with to_asyncio.open_channel_from(
-            partial(
             aio_order_feed_relay,
-                fh,
-                instrument
-            )
+            fh=fh,
+            instrument=instrument,
         ) as (chan, first):
             yield chan
 

piker/brokers/ib/__init__.py

@@ -65,17 +65,18 @@ _brokerd_mods: list[str] = [
 ]
 
 _datad_mods: list[str] = [
+    'api',
     'feed',
     'symbols',
 ]
 
 
 # tractor RPC enable arg
-__enable_modules__: list[str] = (
+__enable_modules__: list[str] = list(dict.fromkeys(
     _brokerd_mods
     +
     _datad_mods
-)
+))
 
 # passed to ``tractor.ActorNursery.start_actor()``
 _spawn_kwargs = {

piker/brokers/ib/api.py

@@ -95,6 +95,7 @@ from .symbols import (
 )
 from ...log import get_logger
 from .venues import (
+    is_expired,
     is_venue_open,
     sesh_times,
     is_venue_closure,
@@ -496,7 +497,7 @@ class Client:
                 await self.ib.reqContractDetailsAsync(contract)
             )[0]
             # convert to makt-native tz
-            tz: str = details.timeZoneId
+            tz: str = details.timeZoneId or 'EST'
             end_dt = end_dt.in_tz(tz)
             first_dt: DateTime = from_timestamp(first).in_tz(tz)
             last_dt: DateTime = from_timestamp(last).in_tz(tz)
@@ -508,10 +509,18 @@ class Client:
             _open_now: bool = is_venue_open(
                 con_deats=details,
             )
+            _is_expired: bool = is_expired(
+                con_deats=details,
+            )
 
             # XXX, do gap detections.
             has_closure_gap: bool = False
             if (
+                # XXX, expired tracts can't be introspected
+                # for open/closure intervals due to ib's chitty
+                # details seemingly..
+                not _is_expired
+                and
                 last_dt.add(seconds=sample_period_s)
                 <
                 end_dt
@@ -958,8 +967,26 @@ class Client:
 
                 else:
                     raise
+            # XXX: ambiguous/unqualifiable contracts are
+            # returned as `None` entries by
+            # `qualifyContractsAsync()` (which also logs an
+            # "Ambiguous contract" warning listing the
+            # possible matches) so filter them BEFORE any
+            # attr access B)
+            contracts: list[Contract] = [
+                tract
+                for tract in contracts
+                if tract is not None
+            ]
             if not contracts:
-                raise ValueError(f"No contract could be found {con}")
+                raise ValueError(
+                    f'No (unambiguous) contract could be '
+                    f'qualified for {con!r}\n'
+                    f'\n'
+                    f'If a stonk, you likely need a (more) '
+                    f'venue-qualified fqme,\n'
+                    f"eg. 'gld.arca.ib' instead of 'gld.ib'\n"
+                )
 
         # pack all contracts into cache
         for tract in contracts:
@@ -1137,10 +1164,16 @@ class Client:
         try:
             con: Contract = self._contracts[symbol]
         except KeyError:
-            # require that the symbol has been previously cached by
-            # a data feed request - ensure we aren't making orders
-            # against non-known prices.
-            raise RuntimeError("Can not order {symbol}, no live feed?")
+            # require that the mkt's contract has been previously
+            # qualified and cached (see `.symbols.get_mkt_info()`
+            # which is run for any feed-init OR lazily by the
+            # order-request handler in `.broker`) - ensures we
+            # aren't making orders against unknown contracts.
+            raise RuntimeError(
+                f'Can not order {symbol!r}, '
+                f'no qualified contract cached!?\n'
+                f'_contracts: {list(self._contracts)!r}\n'
+            )
 
         try:
             trade = self.ib.placeOrder(
@@ -1331,6 +1364,21 @@ async def load_aio_clients(
     ib = None
     client = None
 
+    # XXX: post (brokerd vs. datad)-split BOTH per-broker
+    # daemons connect to the same API gw/tws endpoint(s); to
+    # avoid `clientId` collisions (and the long conn-timeout
+    # retry cycle they cause) we offset the data-daemon's
+    # default id-range to be disjoint from `brokerd.ib`'s
+    # (which also retries with `client_id + i` increments).
+    if client_id == 6116:  # the default from above
+        aname: str = tractor.current_actor().name
+        if 'datad' in aname:
+            client_id += 16
+        # ad-hoc (test/cli) actors get their own range to
+        # avoid clashing with any live daemon-tree's conns.
+        elif 'brokerd' not in aname:
+            client_id += 32
+
     # attempt to get connection info from config; if no .toml entry
     # exists, we try to load from a default localhost connection.
     localhost = '127.0.0.1'
@@ -1584,6 +1632,17 @@ class MethodProxy:
         self.event_table = event_table
         self._aio_ns = asyncio_ns
 
+        # request-id counter for correlating each method
+        # call to its (eventual) response; necessary since
+        # the (single, shared) chan has NO other ordering
+        # guarantee whenever a caller task is CANCELLED
+        # (eg. by a search-req timeout) after sending its
+        # request but before consuming the response: the
+        # "orphaned" response otherwise gets mis-delivered
+        # to the next caller causing an off-by-one skew of
+        # every result thereafter!
+        self._mids = itertools.count()
+
     async def _run_method(
         self,
         *,
@@ -1597,10 +1656,10 @@ class MethodProxy:
 
         '''
         chan = self.chan
-        await chan.send((meth, kwargs))
+        mid: int = next(self._mids)
+        await chan.send((meth, kwargs, mid))
 
         while not chan.closed():
-            # send through method + ``kwargs: dict`` as pair
             msg = await chan.receive()
 
             # TODO: implement reconnect functionality like
@@ -1610,23 +1669,50 @@ class MethodProxy:
             # except ConnectionError:
             #     self.reset()
 
-            # print(f'NEXT MSG: {msg}')
-
-            # TODO: py3.10 ``match:`` syntax B)
-            if 'result' in msg:
-                res = msg.get('result')
+            match msg:
+                # OUR method-call response B)
+                case {'mid': resp_mid, 'result': res} if (
+                    resp_mid == mid
+                ):
                     return res
 
-            elif 'exception' in msg:
-                err = msg.get('exception')
+                case {'mid': resp_mid, 'exception': err} if (
+                    resp_mid == mid
+                ):
                     raise err
 
-            elif 'error' in msg:
-                etype, emsg = msg
+                # an "orphaned" response to some prior
+                # (cancelled) caller's request; drop it and
+                # keep waiting for ours.
+                case {'mid': resp_mid}:
+                    log.warning(
+                        f'Dropping stale method-resp,\n'
+                        f'mid: {resp_mid} (ours: {mid})\n'
+                        f'(a prior caller prolly got '
+                        f'cancelled before its resp?)\n'
+                    )
+                    continue
+
+                # out-of-band (inline) client error: raise
+                # to the current caller as before.
+                case {'exception': err}:
+                    raise err
+
+                case ('error', emsg):
                     log.warning(f'IB error relay: {emsg}')
                     continue
 
-            else:
+                # routine (api-farm conn) status events
+                # relayed inline by `Client.inline_errors()`;
+                # not a response to our method call so just
+                # log at info and keep waiting.
+                case ('event', emsg):
+                    log.info(
+                        f'IB status event relay: {emsg}'
+                    )
+                    continue
+
+                case _:
                     log.warning(f'UNKNOWN IB MSG: {msg}')
 
     def status_event(
@@ -1680,22 +1766,31 @@ async def open_aio_client_method_relay(
                 log.info('asyncio `Client` method-proxy SHUTDOWN!')
                 break
 
-            case (meth_name, kwargs):
-                meth_name, kwargs = msg
+            case (meth_name, kwargs, mid):
                 meth = getattr(client, meth_name)
 
                 try:
                     resp = await meth(**kwargs)
-                    # echo the msg back
-                    chan.send_nowait({'result': resp})
+                    # echo the msg back tagged with the
+                    # req-id for caller-side correlation.
+                    chan.send_nowait({
+                        'mid': mid,
+                        'result': resp,
+                    })
 
-                except (
-                    RequestError,
-
-                    # TODO: relay all errors to trio?
-                    # BaseException,
-                ) as err:
-                    chan.send_nowait({'exception': err})
+                # XXX: relay ALL (non-cancel) errors to the
+                # `trio`-side caller (which re-raises in the
+                # `MethodProxy._run_method()` frame) instead
+                # of crashing this relay task and thus the
+                # whole proxy/dialog; critical post the
+                # (datad|brokerd)-split where eg. contract
+                # qualification failures are expected to be
+                # caught per-request by order/warmup code!
+                except Exception as err:
+                    chan.send_nowait({
+                        'mid': mid,
+                        'exception': err,
+                    })
 
             case {'error': content}:
                 chan.send_nowait({'exception': content})

piker/brokers/ib/broker.py

@@ -89,8 +89,9 @@ from .api import (
     MethodProxy,
 )
 from .symbols import (
+    cache_contract,
     con2fqme,
-    # get_mkt_info,
+    get_mkt_info,
 )
 from .ledger import (
     norm_trade_records,
@@ -138,6 +139,7 @@ async def handle_order_requests(
     ems_order_stream: tractor.MsgStream,
     accounts_def: dict[str, str],
     flows: OrderDialogs,
+    proxies: dict[str, MethodProxy],
 
 ) -> None:
 
@@ -180,6 +182,58 @@ async def handle_order_requests(
         if action in {'buy', 'sell'}:
             # validate
             order = BrokerdOrder(**request_msg)
+            fqme: str = order.symbol
+
+            # XXX: lazily qualify and cache the contract for
+            # this mkt since, post the (datad|brokerd)-split,
+            # this trading-only actor will NOT have had its
+            # (api) `Client._contracts` pre-warmed by any
+            # in-proc feed setup (which now runs in the
+            # `datad.ib` sibling); we run the SAME symbology
+            # resolution ep the feed-side uses so the cache
+            # key (`MktPair.bs_fqme`) matches what
+            # `Client.submit_limit()` looks up.
+            # NOTE: normally a no-op since
+            # `open_trade_dialog()` eagerly pre-qualifies all
+            # already-open pp/order mkts at startup; this
+            # only fires for a brand-new (to this daemon)
+            # mkt's first order.
+            if fqme not in client._contracts:
+                proxy: MethodProxy = proxies[account]
+                try:
+                    (
+                        mkt,
+                        details,
+                    ) = await get_mkt_info(
+                        fqme,
+                        proxy=proxy,
+                    )
+                    # XXX: explicit write since the lifo-cache
+                    # may deliver a hit (body skipped) when
+                    # another acct/client already qualified
+                    # this fqme.
+                    cache_contract(
+                        client,
+                        mkt,
+                        details.contract,
+                    )
+                except Exception as err:
+                    log.exception(
+                        f'Failed to qualify contract for\n'
+                        f'fqme: {fqme!r}\n'
+                    )
+                    await ems_order_stream.send(
+                        BrokerdError(
+                            oid=oid,
+                            symbol=fqme,
+                            reason=(
+                                f'No contract could be '
+                                f'qualified for {fqme!r}:\n'
+                                f'{err!r}'
+                            ),
+                        )
+                    )
+                    continue
 
             # XXX: by default 0 tells ``ib_async`` methods that
             # there is no existing order so ask the client to create
@@ -191,15 +245,34 @@ async def handle_order_requests(
                 reqid = int(reqid)
 
             # call our client api to submit the order
+            # NOTE: guard with order-error relay (vs. crashing
+            # this dialog and thus ALL order ctl for the
+            # backend) so one bad submission can't take down
+            # the daemon's clearing loop.
+            try:
                 reqid = client.submit_limit(
                     oid=order.oid,
-                symbol=order.symbol,
+                    symbol=fqme,
                     price=order.price,
                     action=order.action,
                     size=order.size,
                     account=acct_number,
                     reqid=reqid,
                 )
+            except Exception as err:
+                log.exception(
+                    f'Order submission failed for\n'
+                    f'fqme: {fqme!r}\n'
+                )
+                await ems_order_stream.send(
+                    BrokerdError(
+                        oid=oid,
+                        symbol=fqme,
+                        reason=f'Submission error: {err!r}',
+                    )
+                )
+                continue
+
             str_reqid: str = str(reqid)
             if reqid is None:
                 err_msg = BrokerdError(
@@ -231,20 +304,21 @@ async def handle_order_requests(
 
 async def recv_trade_updates(
 
+    chan: tractor.to_asyncio.LinkedTaskChannel,
     client: Client,
-    to_trio: trio.abc.SendChannel,
 
 ) -> None:
     '''
-    Receive and relay order control and positioning related events
-    from `ib_async`, pack as tuples and push over mem-chan to our
-    trio relay task for processing and relay to EMS.
+    Receive and relay order control and positioning
+    related events from `ib_async`, pack as tuples and
+    push over mem-chan to our trio relay task for
+    processing and relay to EMS.
 
     '''
-    client.inline_errors(to_trio)
+    client.inline_errors(chan)
 
     # sync with trio task
-    to_trio.send_nowait(client.ib)
+    chan.started_nowait(client.ib)
 
     def push_tradesies(
         eventkit_obj,
@@ -282,7 +356,7 @@ async def recv_trade_updates(
 
         try:
             # emit event name + relevant ibis internal objects
-            to_trio.send_nowait((event_name, emit))
+            chan.send_nowait((event_name, emit))
         except trio.BrokenResourceError:
             log.exception(f'Disconnected from {eventkit_obj} updates')
             eventkit_obj.disconnect(push_tradesies)
@@ -547,10 +621,14 @@ async def open_trade_dialog(
 
 ) -> AsyncIterator[dict[str, Any]]:
 
-    get_console_log(
-        level=loglevel,
-        name=__name__,
-    )
+    # XXX: ONLY adjust the level of this (sub)mod's logger;
+    # attaching a (stderr) handler (via `get_console_log()`)
+    # here would DOUBLE-print every record since the daemon
+    # fixture (`.._daemon._setup_persistent_brokerd()`)
+    # already enables the console handler on the parent
+    # subsys logger which all our records propagate to!
+    if loglevel:
+        log.setLevel(loglevel.upper())
 
     # task local msg dialog tracking
     flows = OrderDialogs()
@@ -787,6 +865,64 @@ async def open_trade_dialog(
                 for msg in order_msgs:
                     await ems_stream.send(msg)
 
+                # XXX: eagerly pre-qualify (and cache) the
+                # contracts for all already-open pps and
+                # orders so that (live) order submission
+                # NEVER pays a first-request qualification
+                # delay; any brand-new mkt is still lazily
+                # qualified by `handle_order_requests()` on
+                # its first submission. NOTE: since this
+                # runs BEFORE the order handler task is
+                # even started, no order can clear until
+                # the warmup completes (early reqs just
+                # buffer in the ems IPC stream) B)
+                warmup_fqmes: set[str] = {
+                    msg.symbol
+                    for msg in all_positions
+                }
+                warmup_fqmes.update(
+                    msg.req.symbol
+                    for msg in order_msgs
+                )
+                unique_clients: set[Client] = set(
+                    aioclients.values()
+                )
+                if (
+                    warmup_fqmes
+                    and
+                    proxies
+                ):
+                    a_proxy: MethodProxy = next(
+                        iter(proxies.values())
+                    )
+                    for fqme in warmup_fqmes:
+                        try:
+                            (
+                                mkt,
+                                details,
+                            ) = await get_mkt_info(
+                                fqme,
+                                proxy=a_proxy,
+                            )
+                        except Exception as err:
+                            # XXX: non-fatal; an
+                            # un-warmed mkt just falls
+                            # back to the lazy qualify
+                            # in the order handler.
+                            log.warning(
+                                f'Failed to pre-qualify\n'
+                                f'fqme: {fqme!r}\n'
+                                f'err: {err!r}\n'
+                            )
+                            continue
+
+                        for _client in unique_clients:
+                            cache_contract(
+                                _client,
+                                mkt,
+                                details.contract,
+                            )
+
                 for client in set(aioclients.values()):
                     trade_event_stream: LinkedTaskChannel = await tn.start(
                         open_trade_event_stream,
@@ -800,6 +936,7 @@ async def open_trade_dialog(
                         ems_stream,
                         accounts_def,
                         flows,
+                        proxies,
                     )
 
                     # allocate event relay tasks for each client connection
@@ -1162,9 +1299,9 @@ async def deliver_trade_events(
             case 'execDetailsEvent':
                 # unpack attrs pep-0526 style.
                 trade: Trade
-                con: Contract = trade.contract
                 fill: Fill
                 trade, fill = item
+                con: Contract = trade.contract
                 execu: Execution = fill.execution
                 execid: str = execu.execId
                 report: CommissionReport = fill.commissionReport
@@ -1306,7 +1443,15 @@ async def deliver_trade_events(
                 elif isinstance(err, str):
                     code_part, _, reason = err.rpartition(']')
                     if code_part:
-                        _, _, code = code_part.partition('[code')
+                        for prefix_patt in [
+                            '[Errno ',
+                            '[code ',
+                        ]:
+                            code_part, _, code = code_part.partition()
+                            if code:
+                                code = int(code)
+                                break
+
                     reqid: str = '<unknown>'
 
                 # "Warning:" msg codes,
@@ -1325,6 +1470,8 @@ async def deliver_trade_events(
 
                 # XXX known special (ignore) cases
                 elif code in {
+                    # ^TODO, if this is it we should definitely raise
+                    # or at least provide for reconnect attempts?
                     200,  # uhh.. ni idea
 
                     # hist pacing / connectivity
@@ -1335,10 +1482,18 @@ async def deliver_trade_events(
                     # 'No market data during competing live session'
                     1669,
                 }:
+                    pcc: str = "Peer closed connection."
+                    if pcc in err:
+                        # TODO, emit and raise?
+                        # [ ] try for reconnect?
+                        # raise BrokerdError(
+                        await tractor.pause()
+
                     log.error(
                         f'Order-API-error which is non-cancel-causing ?!\n'
                         f'\n'
-                        f'{pformat(err)}\n'
+                        f'code={code!r}\n'
+                        f'err={pformat(err)}\n'
                     )
                     continue
 

piker/brokers/ib/ledger.py

@@ -501,7 +501,7 @@ async def update_ledger_from_api_trades(
     for fill in fills:
         con: Contract = fill.contract
         conid: str = con.conId
-        pexch: str | None = con.primaryExchange
+        pexch: str|None = con.primaryExchange or con.exchange
 
         if not pexch:
             cons = await client.get_con(conid=conid)

piker/brokers/ib/symbols.py

@@ -639,12 +639,40 @@ async def get_mkt_info(
 
     # if possible register the bs_mktid to the just-built
     # mkt so that it can be retreived by order mode tasks later.
-    # TODO NOTE: this is going to be problematic if/when we split
-    # out the datatd vs. brokerd actors since the mktmap lookup
-    # table will now be inaccessible..
     if proxy is not None:
-        client: Client = proxy._aio_ns
-        client._contracts[mkt.bs_fqme] = con
-        client._cons2mkts[con] = mkt
+        cache_contract(
+            proxy._aio_ns,
+            mkt,
+            con,
+        )
 
     return mkt, details
+
+
+def cache_contract(
+    client: Client,
+    mkt: MktPair,
+    con: ibis.Contract,
+
+) -> None:
+    '''
+    Register a (qualified) contract + mkt-info pair on the
+    given (api) `Client`'s actor-local lookup tables.
+
+    Cached under BOTH fqme key-forms since consumers vary:
+    - `mkt.bs_fqme` (eg. 'nvda.nasdaq'): read by
+      `Client.submit_limit()` for order requests,
+    - `mkt.fqme` (eg. 'nvda.nasdaq.ib'): read by the
+      fill-time pp-update (symcache-backup-table) path in
+      `.broker.emit_pp_update()`.
+
+    NOTE: post the (datad|brokerd)-actor-split this MUST be
+    run (in the trading actor) for every mkt either eagerly
+    at `.broker.open_trade_dialog()` startup or lazily per
+    order request; there is no in-proc feed setup doing it
+    implicitly anymore!
+
+    '''
+    client._contracts[mkt.bs_fqme] = con
+    client._contracts[mkt.fqme] = con
+    client._cons2mkts[con] = mkt

piker/brokers/ib/venues.py

@@ -33,13 +33,21 @@ from typing import (
 )
 
 import exchange_calendars as xcals
+from exchange_calendars.errors import (
+    InvalidCalendarName,
+)
 from pendulum import (
+    parse,
     now,
     Duration,
     Interval,
     Time,
 )
 
+from piker.log import get_logger
+
+log = get_logger(__name__)
+
 if TYPE_CHECKING:
     from ib_async import (
         TradingSession,
@@ -56,6 +64,22 @@ if TYPE_CHECKING:
     )
 
 
+def is_expired(
+    con_deats: ContractDetails,
+) -> bool:
+    '''
+    Simple predicate whether the provided contract-deats match and
+    already lifetime-terminated instrument.
+
+    '''
+    expiry_str: str = con_deats.realExpirationDate
+    if not expiry_str:
+        return False
+
+    expiry_dt: datetime = parse(expiry_str)
+    return expiry_dt.date() >= now().date()
+
+
 def has_weekend(
     period: Interval,
 ) -> bool:
@@ -90,13 +114,28 @@ def has_holiday(
         con.exchange
     )
 
-    # XXX, ad-hoc handle any IB exchange which are non-std
-    # via lookup table..
-    std_exch: dict = {
+    # XXX, ad-hoc handle any IB exchange which are
+    # non-std via lookup table..
+    std_exch: str = {
         'ARCA': 'ARCX',
     }.get(exch, exch)
 
-    cal: ExchangeCalendar = xcals.get_calendar(std_exch)
+    try:
+        cal: ExchangeCalendar = xcals.get_calendar(
+            std_exch
+        )
+    except InvalidCalendarName:
+        # venue has no `exchange_calendars` entry
+        # (eg. IDEALPRO for forex, PAXOS for
+        # crypto) -> not a holiday by default since
+        # weekends are already handled by
+        # `has_weekend()`.
+        log.warning(
+            f'No exchange cal for {std_exch!r},'
+            f' skipping holiday check..\n'
+        )
+        return False
+
     end: datetime = period.end
     # _start: datetime = period.start
     # ?TODO, can rm ya?
@@ -170,7 +209,22 @@ def sesh_times(
     get the (day-agnostic) times for the start/end.
 
     '''
-    earliest_sesh: Interval = next(iter_sessions(con_deats))
+    # ?TODO, lookup the next front contract instead?
+    if is_expired(con_deats):
+        raise ValueError(
+            f'Contract is already expired!\n'
+            f'Choose an active alt contract instead.\n'
+            f'con_deats: {con_deats!r}\n'
+        )
+
+    maybe_sessions: list[Interval] = list(iter_sessions(con_deats))
+    if not maybe_sessions:
+        raise ValueError(
+            f'Contract has no trading-session info?\n'
+            f'con_deats: {con_deats!r}\n'
+        )
+
+    earliest_sesh: Interval = maybe_sessions[0]
     return (
         earliest_sesh.start.time(),
         earliest_sesh.end.time(),
@@ -211,7 +265,13 @@ def is_venue_closure(
     '''
     open: Time
     close: Time
-    open, close = sesh_times(con_deats)
+    maybe_oc: tuple|None = sesh_times(con_deats)
+    if maybe_oc is None:
+        # XXX, should never get here.
+        breakpoint()
+        return False
+
+    open, close = maybe_oc
 
     # ensure times are in mkt-native timezone
     tz: str = con_deats.timeZoneId

piker/brokers/kraken/__init__.py

@@ -66,10 +66,24 @@ __all__ = [
 ]
 
 
-# tractor RPC enable arg
-__enable_modules__: list[str] = [
+# per-daemon-kind (sub)mod groups: declares which of our
+# submods host the eps run by each daemon-actor kind as
+# defined by `piker.data.validate._eps`.
+_brokerd_mods: list[str] = [
     'api',
     'broker',
+]
+
+_datad_mods: list[str] = [
+    'api',
     'feed',
     'symbols',
 ]
+
+
+# tractor RPC enable arg
+__enable_modules__: list[str] = list(dict.fromkeys(
+    _brokerd_mods
+    +
+    _datad_mods
+))

piker/brokers/kraken/api.py

@@ -35,6 +35,7 @@ import hashlib
 import hmac
 import base64
 import tractor
+# from tractor._exceptions import reg_err_types
 import trio
 
 from piker import config
@@ -52,6 +53,7 @@ from piker.brokers._util import (
     SymbolNotFound,
     BrokerError,
     DataThrottle,
+    get_or_raise_on_pair_schema_mismatch,
 )
 from piker.accounting import Transaction
 from piker.log import get_logger
@@ -107,15 +109,37 @@ def get_kraken_signature(
     return sigdigest.decode()
 
 
-class InvalidKey(ValueError):
-    '''
-    EAPI:Invalid key
-    This error is returned when the API key used for the call is
-    either expired or disabled, please review the API key in your
-    Settings -> API tab of account management or generate a new one
-    and update your application.
+# class InvalidKey(ValueError):
+#     '''
+#     EAPI:Invalid key
 
-    '''
+#     This error is returned when the API key used for the call is
+#     either expired or disabled, please review the API key in your
+#     Settings -> API tab of account management or generate a new one
+#     and update your application.
+
+#     '''
+
+
+# class InvalidSession(RuntimeError):
+#     '''
+#     ESession:Invalid session
+
+#     This error is returned when the ws API key used for an authenticated
+#     sub/endpoint becomes stale, normally after a sufficient network
+#     disconnect/outage.
+
+#     Normally the sub will need to be restarted, likely re-init of the
+#     auth handshake sequence.
+
+#     '''
+#     subscription: dict
+
+
+# reg_err_types([
+#     InvalidKey,
+#     InvalidSession,
+# ])
 
 
 class Client:
@@ -143,18 +167,16 @@ class Client:
         config: dict[str, str],
         httpx_client: httpx.AsyncClient,
 
-        name: str = '',
+        key_descr: str = '',
         api_key: str = '',
         secret: str = ''
     ) -> None:
-
         self._sesh: httpx.AsyncClient = httpx_client
-
-        self._name = name
+        self._key_descr = key_descr
         self._api_key = api_key
         self._secret = secret
-
         self.conf: dict[str, str] = config
+        self._ws_token: str|None = None
 
     @property
     def pairs(self) -> dict[str, Pair]:
@@ -239,6 +261,39 @@ class Client:
 
         return balances
 
+    async def get_ws_token(
+        self,
+        params: dict = {},
+        force_renewal: bool = False,
+    ) -> str:
+        '''
+        Get websocket token for authenticated data stream and cache
+        it for reuse.
+
+        Assert a value was actually received before return.
+
+        '''
+        if (
+            not force_renewal
+            and
+            self._ws_token
+        ):
+            return self._ws_token
+
+        resp = await self.endpoint(
+            'GetWebSocketsToken',
+            {},
+        )
+        if err := resp.get('error'):
+            raise BrokerError(err)
+
+        # resp token for ws init
+        token: str = resp['result']['token']
+        assert token
+
+        self._ws_token: str = token
+        return token
+
     async def get_assets(
         self,
         reload: bool = False,
@@ -502,7 +557,16 @@ class Client:
 
                 # NOTE: always cache in pairs tables for faster lookup
                 with tractor.devx.maybe_open_crash_handler(): # as bxerr:
-                    pair = Pair(xname=xkey, **data)
+                    # pair = Pair(xname=xkey, **data)
+                    pair: Pair = get_or_raise_on_pair_schema_mismatch(
+                        pair_type=Pair,
+                        fields_data=dict(
+                            xname=xkey,
+                            **data,
+                        ),
+                        provider_name='kraken',
+                        # api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
+                    )
 
                 # register the above `Pair` structs for all
                 # key-sets/monikers: a set of 4 (frickin) tables
@@ -668,7 +732,13 @@ class Client:
 
 @acm
 async def get_client() -> Client:
+    '''
+    Load and deliver a `.kraken.api.Client`.
 
+    When defined, inject any config delivered from the user's
+    `brokers.toml` config file.
+
+    '''
     conf: dict[str, Any] = get_config()
     async with httpx.AsyncClient(
         base_url=_url,
@@ -679,13 +749,14 @@ async def get_client() -> Client:
         # connections=4
     ) as trio_client:
         if conf:
+            api_key_descr: str = conf['key_descr']
             client = Client(
                 conf,
                 httpx_client=trio_client,
 
                 # TODO: don't break these up and just do internal
                 # conf lookups instead..
-                name=conf['key_descr'],
+                key_descr=api_key_descr,
                 api_key=conf['api_key'],
                 secret=conf['secret']
             )

piker/brokers/kraken/broker.py

@@ -30,12 +30,15 @@ from typing import (
     Any,
     AsyncIterator,
     Iterable,
+    Type,
     Union,
 )
 
 from bidict import bidict
 import trio
 import tractor
+from tractor.devx.pformat import ppfmt
+# from tractor._exceptions import reg_err_types
 
 from piker.accounting import (
     Position,
@@ -45,6 +48,9 @@ from piker.accounting import (
     open_trade_ledger,
     open_account,
 )
+from piker.config import (
+    ConfigurationError,
+)
 from piker.clearing import(
     OrderDialogs,
 )
@@ -67,13 +73,12 @@ from piker.log import (
     get_logger,
 )
 from piker.data import open_symcache
-from .api import (
-    Client,
-    BrokerError,
-)
-from .feed import (
+from piker.data._web_bs import (
     open_autorecon_ws,
     NoBsWs,
+)
+from . import api
+from .feed import (
     stream_messages,
 )
 from .ledger import (
@@ -94,11 +99,7 @@ MsgUnion = Union[
 ]
 
 
-class TooFastEdit(Exception):
-    'Edit requests faster then api submissions'
-
-
-# TODO: make this wrap the `Client` and `ws` instances
+# TODO: make this wrap the `api.Client` and `ws` instances
 # and give it methods to submit cancel vs. add vs. edit
 # requests?
 class BrokerClient:
@@ -126,23 +127,22 @@ class BrokerClient:
 
 
 async def handle_order_requests(
-
     ws: NoBsWs,
-    client: Client,
+    client: api.Client,
     ems_order_stream: tractor.MsgStream,
-    token: str,
     apiflows: OrderDialogs,
     ids: bidict[str, int],
     reqids2txids: dict[int, str],
+    toofastedit: set[int],
 
 ) -> None:
     '''
-    Process new order submission requests from the EMS
-    and deliver acks or errors.
+    `trio.Task` which handles order ctl requests from the EMS and
+    deliver acks or errors back on that IPC dialog.
 
     '''
     # XXX: UGH, let's unify this.. with ``msgspec``!!!
-    msg: dict | Order
+    msg: dict|Order
     async for msg in ems_order_stream:
         log.info(f'Rx order msg:\n{pformat(msg)}')
         match msg:
@@ -156,8 +156,13 @@ async def handle_order_requests(
                     txid = reqids2txids[reqid]
                 except KeyError:
                     # XXX: not sure if this block ever gets hit now?
+                    # SEEMS TO on the race case with the update task?
+                    # - update dark order quickly after
+                    #   triggered-submitted and then we have inavlid
+                    #   value in `reqids2txids` sent over ws.send()??
                     log.error('TOO FAST CANCEL/EDIT')
-                    reqids2txids[reqid] = TooFastEdit(reqid)
+                    toofastedit.add(reqid)
+                    reqids2txids[reqid] = reqid
                     await ems_order_stream.send(
                         BrokerdError(
                             oid=msg['oid'],
@@ -173,7 +178,7 @@ async def handle_order_requests(
                     # https://docs.kraken.com/websockets/#message-cancelOrder
                     await ws.send_msg({
                         'event': 'cancelOrder',
-                        'token': token,
+                        'token': await client.get_ws_token(),
                         'reqid': reqid,
                         'txid': [txid],  # should be txid from submission
                     })
@@ -185,7 +190,7 @@ async def handle_order_requests(
                 # validate
                 order = BrokerdOrder(**msg)
 
-                # logic from old `Client.submit_limit()`
+                # logic from old `api.Client.submit_limit()`
                 if order.oid in ids:
                     ep: str = 'editOrder'
                     reqid: int = ids[order.oid]  # integer not txid
@@ -195,13 +200,15 @@ async def handle_order_requests(
 
                         # XXX: not sure if this block ever gets hit now?
                         log.error('TOO FAST EDIT')
-                        reqids2txids[reqid] = TooFastEdit(reqid)
+                        reqids2txids[reqid] = reqid
+                        toofastedit.add(reqid)
+                        await tractor.pause()
                         await ems_order_stream.send(
                             BrokerdError(
                                 oid=msg['oid'],
                                 symbol=msg['symbol'],
                                 reason=(
-                                    f'TooFastEdit reqid:{reqid}, cancelling..'
+                                    f'TooFastEdit reqid: {reqid}, cancelling..'
                                 ),
 
                             )
@@ -247,7 +254,7 @@ async def handle_order_requests(
                 # https://docs.kraken.com/websockets/#message-addOrder
                 req = {
                     'event': ep,
-                    'token': token,
+                    'token': await client.get_ws_token(),
 
                     'reqid': reqid,  # remapped-to-int uid from ems
                     # XXX: we set these to the same value since for us
@@ -291,13 +298,15 @@ async def handle_order_requests(
                         symbol=msg['symbol'],
                         reason=(
                             'Invalid request msg:\n{msg}'
-                        ))
+                        ),
+                    )
                 )
 
 
 @acm
 async def subscribe(
     ws: NoBsWs,
+    client: api.Client,
     token: str,
     subs: list[tuple[str, dict]] = [
         ('ownTrades', {
@@ -316,12 +325,25 @@ async def subscribe(
     Setup ws api subscriptions:
     https://docs.kraken.com/websockets/#message-subscribe
 
-    By default we sign up for trade and order update events.
+    By default we sign up for trade and order (update) events per
+    `subs`.
 
     '''
     # more specific logic for this in kraken's sync client:
     # https://github.com/krakenfx/kraken-wsclient-py/blob/master/kraken_wsclient_py/kraken_wsclient_py.py#L188
-    assert token
+    latest_token: str = await client.get_ws_token()
+    if (
+        token
+        !=
+        latest_token
+    ):
+        log.info(
+            f'RE-subscribing to WS connection..\n'
+            f'orig-token: {token!r}\n'
+            f'latest-token: {latest_token!r}\n'
+        )
+        token = latest_token
+
     subnames: set[str] = set()
 
     for name, sub_opts in subs:
@@ -329,7 +351,8 @@ async def subscribe(
             'event': 'subscribe',
             'subscription': {
                 'name': name,
-                'token': token,
+                # 'token': await client.get_ws_token(),
+                'token': latest_token,
                 **sub_opts,
             }
         }
@@ -344,7 +367,9 @@ async def subscribe(
     # wait on subscriptionn acks
     with trio.move_on_after(5):
         while True:
-            match (msg := await ws.recv_msg()):
+            msg: dict = await ws.recv_msg()
+            fmt_msg: str = ppfmt(msg)
+            match msg:
                 case {
                     'event': 'subscriptionStatus',
                     'status': 'subscribed',
@@ -362,10 +387,49 @@ async def subscribe(
                     'event': 'subscriptionStatus',
                     'status': 'error',
                     'errorMessage': errmsg,
+                    'subscription': sub_opts,
                 } as msg:
-                    raise RuntimeError(
-                        f'{errmsg}\n\n'
-                        f'{pformat(msg)}'
+                    if errmsg:
+                        etype_str, _, ev_msg = errmsg.partition(':')
+                        etype: Type[Exception] = getattr(
+                            api,
+                            etype_str,
+                            RuntimeError,
+                        )
+
+                    exc = etype(
+                        f'{ev_msg}\n'
+                        f'\n'
+                        f'{fmt_msg}'
+                    )
+                    # !TODO, for `InvalidSession` we should
+                    # attempt retries to resub and ensure all
+                    # sibling (task) `token` holders update
+                    # their refs accoridingly!
+                    match (etype_str, ev_msg):
+                        case (
+                            'ESession',
+                            'Invalid session',
+                        ):
+                            # attempt ws-token refresh
+                            token: str = await client.get_ws_token(
+                                force_renewal=True
+                            )
+                            await tractor.pause()
+                            continue
+
+                        case _:
+                            log.warning(
+                                f'Unhandled subscription-status,\n'
+                                f'{fmt_msg}'
+                            )
+
+                    raise exc
+
+                case _:
+                    log.warning(
+                        f'Unknown ws event rxed?\n'
+                        f'{fmt_msg}'
                     )
     yield
 
@@ -461,11 +525,27 @@ async def open_trade_dialog(
         # (much like the web UI let's you set an "account currency")
         # such that all positions (nested or flat) will be translated to
         # this source currency's terms.
-        src_fiat = client.conf['src_fiat']
+        src_fiat = client.conf.get('src_fiat')
+        if not src_fiat:
+            raise ConfigurationError(
+                'No `src_fiat: str` field defined in `brokers.toml`'
+            )
 
         # auth required block
-        acctid = client._name
-        acc_name = 'kraken.' + acctid
+        conf: dict = client.conf
+        accounts: dict = conf.get('accounts')
+        acctid: str = client._key_descr
+        if not accounts.get(acctid):
+            raise ConfigurationError(
+                f'No API-key found for account-alias defined as {acctid!r} !\n'
+                f'\n'
+                f'Did set a `kraken.accounts.*` entry in your `brokers.toml`?\n'
+                f'It should look something like,\n'
+                f'\n'
+                f'[kraken]\n'
+                f'accounts.{acctid} = {acctid!r}\n'
+            )
+        fqan: str = f'kraken.{acctid}'
 
         # task local msg dialog tracking
         apiflows = OrderDialogs()
@@ -584,7 +664,10 @@ async def open_trade_dialog(
                 acctid,
             )
             # sync with EMS delivering pps and accounts
-            await ctx.started((ppmsgs, [acc_name]))
+            await ctx.started((
+                ppmsgs,
+                [fqan],
+            ))
 
             # TODO: ideally this blocks the this task
             # as little as possible. we need to either do
@@ -592,14 +675,11 @@ async def open_trade_dialog(
             # async file IO api?
             acnt.write_config()
 
-            # Get websocket token for authenticated data stream
-            # Assert that a token was actually received.
-            resp = await client.endpoint('GetWebSocketsToken', {})
-            if err := resp.get('error'):
-                raise BrokerError(err)
+            token: str = await client.get_ws_token()
 
-            # resp token for ws init
-            token: str = resp['result']['token']
+            # XXX tracks EMS orders which are updated too quickly
+            # on the emds side with sync-issues on the kraken side.
+            toofastedit: set[int] = set()
 
             ws: NoBsWs
             async with (
@@ -608,23 +688,24 @@ async def open_trade_dialog(
                     'wss://ws-auth.kraken.com/',
                     fixture=partial(
                         subscribe,
+                        client=client,
                         token=token,
                     ),
                 ) as ws,
                 aclosing(stream_messages(ws)) as stream,
-                trio.open_nursery() as nurse,
+                trio.open_nursery() as tn,
             ):
                 # task for processing inbound requests from ems
-                nurse.start_soon(
+                tn.start_soon(partial(
                     handle_order_requests,
-                    ws,
-                    client,
-                    ems_stream,
-                    token,
-                    apiflows,
-                    ids,
-                    reqids2txids,
-                )
+                    ws=ws,
+                    client=client,
+                    ems_order_stream=ems_stream,
+                    apiflows=apiflows,
+                    ids=ids,
+                    reqids2txids=reqids2txids,
+                    toofastedit=toofastedit,
+                ))
 
                 # enter relay loop
                 await handle_order_updates(
@@ -635,22 +716,23 @@ async def open_trade_dialog(
                     apiflows=apiflows,
                     ids=ids,
                     reqids2txids=reqids2txids,
+                    toofastedit=toofastedit,
                     acnt=acnt,
                     ledger=ledger,
                     acctid=acctid,
-                    acc_name=acc_name,
-                    token=token,
+                    acc_name=fqan,
                 )
 
 
 async def handle_order_updates(
-    client: Client,  # only for pairs table needed in ledger proc
+    client: api.Client,  # only for pairs table needed in ledger proc
     ws: NoBsWs,
     ws_stream: AsyncIterator,
     ems_stream: tractor.MsgStream,
     apiflows: OrderDialogs,
     ids: bidict[str, int],
     reqids2txids: bidict[int, str],
+    toofastedit: set[int],
     acnt: Account,
 
     # transaction records which will be updated
@@ -659,7 +741,6 @@ async def handle_order_updates(
     # ledger_trans: dict[str, Transaction],
     acctid: str,
     acc_name: str,
-    token: str,
 
 ) -> None:
     '''
@@ -789,7 +870,7 @@ async def handle_order_updates(
                 for order_msg in order_msgs:
                     log.info(
                         f'`openOrders` msg update_{seq}:\n'
-                        f'{pformat(order_msg)}'
+                        f'{ppfmt(order_msg)}'
                     )
                     txid, update_msg = list(order_msg.items())[0]
 
@@ -959,10 +1040,8 @@ async def handle_order_updates(
                             # <-> ems dialog.
                             if (
                                 status == 'open'
-                                and isinstance(
-                                    reqids2txids.get(reqid),
-                                    TooFastEdit
-                                )
+                                and
+                                reqid in toofastedit
                             ):
                                 # TODO: don't even allow this case
                                 # by not moving the client side line
@@ -977,7 +1056,8 @@ async def handle_order_updates(
                                 # https://docs.kraken.com/websockets/#message-cancelOrder
                                 await ws.send_msg({
                                     'event': 'cancelOrder',
-                                    'token': token,
+                                    # 'token': token,
+                                    'token': await client.get_ws_token(),
                                     'reqid': reqid or 0,
                                     'txid': [txid],
                                 })
@@ -1123,7 +1203,8 @@ async def handle_order_updates(
                         txid
 
                         # we throttle too-fast-requests on the ems side
-                        and not isinstance(txid, TooFastEdit)
+                        and
+                        reqid in toofastedit
                     ):
                         # client was editting too quickly
                         # so we instead cancel this order
@@ -1131,7 +1212,8 @@ async def handle_order_updates(
                             f'Cancelling {reqid}@{txid} due to:\n {event}')
                         await ws.send_msg({
                             'event': 'cancelOrder',
-                            'token': token,
+                            # 'token': token,
+                            'token': await client.get_ws_token(),
                             'reqid': reqid or 0,
                             'txid': [txid],
                         })

piker/brokers/kraken/symbols.py

@@ -19,6 +19,9 @@ Symbology defs and search.
 
 '''
 from decimal import Decimal
+from typing import (
+    ClassVar,
+)
 
 import tractor
 
@@ -86,9 +89,14 @@ class Pair(Struct):
     short_position_limit: float = 0
     long_position_limit: float = float('inf')
 
+    # TODO, add API note when this was added!
+    execution_venue: str|None = None
+
     # TODO: should we make this a literal NamespacePath ref?
     ns_path: str = 'piker.brokers.kraken:Pair'
 
+    _api_url: ClassVar[str] = 'https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs'
+
     @property
     def bs_mktid(self) -> str:
         '''

piker/brokers/questrade.py

@@ -95,6 +95,9 @@ _time_frames = {
 class QuestradeError(Exception):
     "Non-200 OK response code"
 
+from tractor._exceptions import reg_err_types
+reg_err_types([QuestradeError])
+
 
 class ContractsKey(NamedTuple):
     symbol: str

piker/clearing/_ems.py

@@ -26,7 +26,6 @@ from collections import (
 from contextlib import asynccontextmanager as acm
 from decimal import Decimal
 from math import isnan
-from pprint import pformat
 from time import time_ns
 from types import ModuleType
 from typing import (
@@ -43,6 +42,7 @@ import trio
 from trio_typing import TaskStatus
 import tractor
 from tractor import trionics
+from tractor.devx.pformat import ppfmt
 
 from ._util import (
     log,  # sub-sys logger
@@ -335,9 +335,14 @@ class TradesRelay(Struct):
 @acm
 async def open_brokerd_dialog(
     brokermod: ModuleType,
-    portal: tractor.Portal,
     exec_mode: str,
     fqme: str|None = None,
+
+    # XXX: explicit (already spawned) trading-actor override,
+    # currently only used by the `piker ledger` cli which
+    # boots its own ad-hoc `brokerd`-like actor; normally we
+    # (lazily) spawn/find the `brokerd.<broker>` daemon here.
+    portal: tractor.Portal|None = None,
     loglevel: str|None = None,
 
 ) -> tuple[
@@ -351,6 +356,10 @@ async def open_brokerd_dialog(
     paper engine instance depending on live trading support for the
     broker backend, configuration, or client code usage.
 
+    NOTE: this is now the ONE place where a (live, credentialed)
+    `brokerd.<broker>` daemon-actor gets (lazily) booted; pure
+    data/paper sessions should never spawn one!
+
     '''
     get_console_log(
         level=loglevel,
@@ -416,16 +425,29 @@ async def open_brokerd_dialog(
         )
         exec_mode: str = 'paper'
 
-    if (
-        trades_endpoint is not None
-        or
-        exec_mode != 'paper'
-    ):
-        # open live brokerd trades endpoint
-        open_trades_endpoint = portal.open_context(
-            trades_endpoint,
-            loglevel=loglevel,
+    @acm
+    async def acquire_live_portal():
+        '''
+        Deliver a portal to the (live, credentialed) trading
+        actor hosting the backend's `open_trade_dialog()` ep:
+        either the caller-provided override or the (maybe
+        lazily spawned) `brokerd.<broker>` service daemon.
+
+        '''
+        if portal is not None:
+            yield portal
+            return
+
+        # XXX: the ONE (normal) place a `brokerd.<broker>`
+        # daemon-actor gets booted in the runtime B)
+        from piker.brokers._daemon import (
+            maybe_spawn_brokerd,
         )
+        async with maybe_spawn_brokerd(
+            brokermod.name,
+            loglevel=loglevel,
+        ) as live_portal:
+            yield live_portal
 
     @acm
     async def maybe_open_paper_ep():
@@ -437,7 +459,14 @@ async def open_brokerd_dialog(
                 return
 
         # open trades-dialog endpoint with backend broker
-        async with open_trades_endpoint as msg:
+        async with (
+            acquire_live_portal() as live_portal,
+
+            live_portal.open_context(
+                trades_endpoint,
+                loglevel=loglevel,
+            ) as msg,
+        ):
             ctx, first = msg
 
             # runtime indication that the backend can't support live
@@ -490,7 +519,7 @@ async def open_brokerd_dialog(
             msg = BrokerdPosition(**msg)
             log.info(
                 f'loading pp for {brokermod.__name__}:\n'
-                f'{pformat(msg.to_dict())}',
+                f'{ppfmt(msg.to_dict())}',
             )
 
             # TODO: state any mismatch here?
@@ -581,7 +610,6 @@ class Router(Struct):
     async def maybe_open_brokerd_dialog(
         self,
         brokermod: ModuleType,
-        portal: tractor.Portal,
         exec_mode: str,
         fqme: str,
         loglevel: str,
@@ -606,7 +634,6 @@ class Router(Struct):
 
         async with open_brokerd_dialog(
             brokermod=brokermod,
-            portal=portal,
             exec_mode=exec_mode,
             fqme=fqme,
             loglevel=loglevel,
@@ -668,7 +695,6 @@ class Router(Struct):
             brokername, _, _, _ = unpack_fqme(fqme)
             brokermod = feed.mods[brokername]
             broker = brokermod.name
-            portal = feed.portals[brokermod]
 
             # XXX: this should be initial price quote from target provider
             flume = feed.flumes[fqme]
@@ -682,7 +708,6 @@ class Router(Struct):
 
             async with self.maybe_open_brokerd_dialog(
                 brokermod=brokermod,
-                portal=portal,
                 exec_mode=exec_mode,
                 fqme=fqme,
                 loglevel=loglevel,
@@ -840,7 +865,7 @@ async def translate_and_relay_brokerd_events(
 
     brokerd_msg: dict[str, Any]
     async for brokerd_msg in brokerd_trades_stream:
-        fmsg = pformat(brokerd_msg)
+        fmsg = ppfmt(brokerd_msg)
         log.info(
             f'Rx brokerd trade msg:\n'
             f'{fmsg}'
@@ -1039,6 +1064,7 @@ async def translate_and_relay_brokerd_events(
                     )
 
                 status_msg.reqid = reqid  # THIS LINE IS CRITICAL!
+                if not status_msg.brokerd_msg:
                     status_msg.brokerd_msg = msg
                 status_msg.src = msg.broker_details['name']
 
@@ -1072,7 +1098,7 @@ async def translate_and_relay_brokerd_events(
                 else:  # open
                     # relayed from backend but probably not handled so
                     # just log it
-                    log.info(f'{broker} opened order {msg}')
+                    log.info(f'{broker!r} opened order {msg}')
 
             # BrokerdFill
             case {
@@ -1185,7 +1211,7 @@ async def translate_and_relay_brokerd_events(
             }:
                 msg = (
                     f'Unhandled broker status for dialog {reqid}:\n'
-                    f'{pformat(brokerd_msg)}'
+                    f'{ppfmt(brokerd_msg)}'
                 )
                 if (
                     oid := book._ems2brokerd_ids.inverse.get(reqid)
@@ -1194,7 +1220,7 @@ async def translate_and_relay_brokerd_events(
                     # clearable limits..
                     if status_msg := book._active.get(oid):
                         msg += (
-                            f'last status msg: {pformat(status_msg)}\n\n'
+                            f'last status msg: {ppfmt(status_msg)}\n\n'
                             f'this msg:{fmsg}\n'
                         )
 
@@ -1233,7 +1259,7 @@ async def process_client_order_cmds(
     async for cmd in client_order_stream:
         log.info(
             f'Received order cmd:\n'
-            f'{pformat(cmd)}\n'
+            f'{ppfmt(cmd)}\n'
         )
 
         # CAWT DAMN we need struct support!
@@ -1398,8 +1424,8 @@ async def process_client_order_cmds(
                 # handle relaying the ems side responses back to
                 # the client/cmd sender from this request
                 log.info(
-                    f'Sending live order to {broker}:\n'
-                    f'{pformat(msg)}'
+                    f'Sending live order to {broker!r}:\n'
+                    f'{ppfmt(msg)}'
                 )
 
                 await brokerd_order_stream.send(msg)

piker/cli/__init__.py

@@ -27,7 +27,7 @@ from types import ModuleType
 import click
 import trio
 import tractor
-from tractor._multiaddr import parse_maddr
+from tractor.discovery import parse_endpoints
 
 from ..log import (
     get_console_log,
@@ -46,48 +46,25 @@ log = get_logger('piker.cli')
 
 
 def load_trans_eps(
-    network: dict | None = None,
-    maddrs: list[tuple] | None = None,
+    network: dict|None = None,
+    maddrs: list[str]|None = None,
 
-) -> dict[str, dict[str, dict]]:
+) -> dict[str, list]:
+    '''
+    Load transport endpoints from a `[network]` config
+    table or CLI-provided multiaddr strings, delegating
+    to `tractor.discovery.parse_endpoints()`.
 
-    # transport-oriented endpoint multi-addresses
-    eps: dict[
-        str,  # service name, eg. `pikerd`, `emsd`..
-
-        # libp2p style multi-addresses parsed into prot layers
-        list[dict[str, str | int]]
-    ] = {}
-
-    if (
-        network
-        and
-        not maddrs
-    ):
-        # load network section and (attempt to) connect all endpoints
-        # which are reachable B)
-        for key, maddrs in network.items():
-            match key:
-
-                # TODO: resolve table across multiple discov
-                # prots Bo
-                case 'resolv':
-                    pass
-
-                case 'pikerd':
-                    dname: str = key
-                    for maddr in maddrs:
-                        layers: dict = parse_maddr(maddr)
-                        eps.setdefault(
-                            dname,
-                            [],
-                        ).append(layers)
+    '''
+    if network and not maddrs:
+        return parse_endpoints(network)
 
     elif maddrs:
-        # presume user is manually specifying the root actor ep.
-        eps['pikerd'] = [parse_maddr(maddr)]
+        return parse_endpoints(
+            {'pikerd': list(maddrs)}
+        )
 
-    return eps
+    return {}
 
 
 @click.command()
@@ -108,13 +85,13 @@ def load_trans_eps(
     help='Enable tractor debug mode',
 )
 @click.option(
-    '--maddr',
+    '--maddrs',
     '-m',
     default=None,
     help='Multiaddrs to bind or contact',
 )
 def pikerd(
-    maddr: list[str] | None,
+    maddrs: list[str]|None,
     loglevel: str,
     tl: bool,
     pdb: bool,
@@ -145,7 +122,8 @@ def pikerd(
         ))
 
     # service-actor registry endpoint socket-address set
-    regaddrs: list[tuple[str, int]] = []
+    regaddrs: list = []
+    tpt_bind_addrs: list|None = None
 
     conf, _ = config.load(
         conf_name='conf',
@@ -153,7 +131,8 @@ def pikerd(
     network: dict = conf.get('network')
     if (
         network is None
-        and not maddr
+        and
+        not maddrs
     ):
         regaddrs = [(
             _default_registry_host,
@@ -161,15 +140,23 @@ def pikerd(
         )]
 
     else:
-        eps: dict = load_trans_eps(
+        from tractor.devx import maybe_open_crash_handler
+        with maybe_open_crash_handler(pdb=pdb):
+            eps: dict[str, list] = load_trans_eps(
                 network,
-            maddr,
+                maddrs,
+            )
+
+            # pikerd bind addresses (may differ from
+            # registry if regd is separate)
+            tpt_bind_addrs = eps.get('pikerd')
+
+            # registry: dedicated `regd` key, or fall
+            # back to pikerd addrs (pikerd IS the registry)
+            regaddrs = eps.get(
+                'regd',
+                tpt_bind_addrs,
             )
-        for layers in eps['pikerd']:
-            regaddrs.append((
-                layers['ipv4']['addr'],
-                layers['tcp']['port'],
-            ))
 
     from .. import service
 
@@ -178,6 +165,7 @@ def pikerd(
         async with (
             service.open_pikerd(
                 registry_addrs=regaddrs,
+                tpt_bind_addrs=tpt_bind_addrs,
                 loglevel=loglevel,
                 debug_mode=pdb,
                 # enable_transports=['uds'],
@@ -262,7 +250,6 @@ def cli(
 
     # TODO: load endpoints from `conf::[network].pikerd`
     # - pikerd vs. regd, separate registry daemon?
-    # - expose datad vs. brokerd?
     # - bind emsd with certain perms on public iface?
     regaddrs: list[tuple[str, int]] = regaddr or [(
         _default_registry_host,
@@ -345,7 +332,7 @@ def services(
     if not ports:
         ports: list[int] = [_default_registry_port]
 
-    addr = tractor._addr.wrap_address(
+    addr = tractor.discovery._addr.wrap_address(
         addr=(host, ports[0])
     )
 

piker/config.py

@@ -36,6 +36,8 @@ except ModuleNotFoundError:
     import tomli as tomllib
 
 
+from tractor._exceptions import reg_err_types
+from tractor.devx.pformat import ppfmt
 from .log import get_logger
 
 log = get_logger('broker-config')
@@ -172,6 +174,12 @@ class ConfigurationError(Exception):
 class NoSignature(ConfigurationError):
     'No credentials setup for broker backend!'
 
+# auto-register for tractor IPC exc-marshalling.
+reg_err_types([
+    ConfigurationError,
+    *ConfigurationError.__subclasses__(),
+])
+
 
 def _override_config_dir(
     path: str
@@ -180,6 +188,48 @@ def _override_config_dir(
     _config_dir = path
 
 
+def _maybe_use_test_dir() -> None:
+    '''
+    When running under the `pytest` harness, override
+    the config dir to the per-test-tmp dir "passed down"
+    the actor tree via `tractor`'s runtime-vars
+    inheritance mechanism.
+
+    See the `tractor_runtime_overrides` usage in our
+    `tests.conftest._open_test_pikerd()` as well as
+    `.service._actor_runtime.open_piker_runtime()` for
+    the root-actor's pre-loading of the var state.
+
+    NOTE: this must be checked lazily at config-path
+    access time (NOT import time) since sub-actors only
+    receive runtime-vars once their `tractor` runtime
+    has fully booted.
+
+    '''
+    global _config_dir
+    import tractor
+    actor = tractor.current_actor(
+        err_on_no_runtime=False,
+    )
+    if actor is None:
+        return
+
+    rvs: dict = tractor.runtime._state._runtime_vars
+    pvars: dict|None = rvs.get('piker_vars')
+    if (
+        pvars
+        and
+        (testdir := pvars.get('piker_test_dir'))
+    ):
+        testdirpath = Path(testdir)
+        assert testdirpath.exists(), (
+            f'piker test harness might be borked!?\n'
+            f'testdirpath: {testdirpath!r}\n'
+        )
+        if _config_dir != testdirpath:
+            _override_config_dir(testdirpath)
+
+
 def _conf_fn_w_ext(
     name: str,
 ) -> str:
@@ -193,6 +243,7 @@ def get_conf_dir() -> Path:
     on the local filesystem.
 
     '''
+    _maybe_use_test_dir()
     return _config_dir
 
 
@@ -218,7 +269,7 @@ def get_conf_path(
         assert str(conf_name) in _conf_names
 
     fn = _conf_fn_w_ext(conf_name)
-    return _config_dir / Path(fn)
+    return get_conf_dir() / Path(fn)
 
 
 def repodir() -> Path:
@@ -263,8 +314,9 @@ def load(
 
     '''
     # create the $HOME/.config/piker dir if dne
-    if not _config_dir.is_dir():
-        _config_dir.mkdir(
+    conf_dir: Path = get_conf_dir()
+    if not conf_dir.is_dir():
+        conf_dir.mkdir(
             parents=True,
             exist_ok=True,
         )
@@ -350,30 +402,56 @@ def write(
 
 
 def load_accounts(
-    providers: list[str] | None = None
-
-) -> bidict[str, str | None]:
+    providers: list[str]|None = None
+) -> bidict[str, str|None]:
 
     conf, path = load(
         conf_name='brokers',
     )
-    accounts = bidict()
-    for provider_name, section in conf.items():
-        accounts_section = section.get('accounts')
-        if (
-            providers is None or
-            providers and provider_name in providers
-        ):
+    accounts = bidict({
+        # XXX, default paper-engine entry; this MUST be set.
+        'paper': None,
+    })
+    msg: str = (
+        'Loading account(s) from `brokers.toml`,\n'
+    )
+    for (
+        provider_name,
+        section,
+    ) in conf.items():
+        accounts_section: dict[str, str] = section.get('accounts')
         if accounts_section is None:
-                log.warning(f'No accounts named for {provider_name}?')
+            msg += f'No accounts declared for {provider_name!r}?\n'
             continue
+
+        # msg += f'Loaded accounts for {provider_name!r}?\n'
+        if (
+            providers is None
+            or (
+                providers
+                and
+                provider_name in providers
+            )
+        ):
+            for (
+                label,
+                value,
+            ) in accounts_section.items():
+                account_alias: str = f'{provider_name}.{label}'
+                accounts[account_alias] = value
+                msg += f'{account_alias} = {value!r}\n'
+
         else:
-                for label, value in accounts_section.items():
-                    accounts[
-                        f'{provider_name}.{label}'
-                    ] = value
-
-    # our default paper engine entry
-    accounts['paper'] = None
+            log.debug(
+                f'NOT loading account(s) for entry in `brokers.toml`,\n'
+                f'The account provider was not requested for loading.\n'
+                f'requested-providers: {providers!r}\n'
+                f'this-provider: {provider_name!r}\n'
+                f'\n'
+                f'{ppfmt(accounts_section)}\n'
+            )
+            # ?TODO? mk this bp work?
+            # breakpoint()
 
+    log.info(msg)
     return accounts

piker/data/_daemon.py

@@ -0,0 +1,312 @@
+# piker: trading gear for hackers
+# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Data-daemon-actor "endpoint-hooks": the service task entry
+points for `datad.<brokername>`, the per-provider real-time
+and historical market-data feed daemon.
+
+The (data vs. broker)d-split mirrors the ep-groupings in
+`piker.data.validate._eps`: this daemon hosts all `'datad'`
+eps (live quotes, history loading, symbology search) while
+its `brokerd.<brokername>` sibling hosts only the live
+order-control (and thus credentialed) `'brokerd'` eps.
+
+'''
+from __future__ import annotations
+from contextlib import (
+    asynccontextmanager as acm,
+)
+from types import ModuleType
+from typing import (
+    TYPE_CHECKING,
+    AsyncContextManager,
+)
+
+import tractor
+import trio
+
+from piker.log import (
+    get_logger,
+    get_console_log,
+)
+from . import _util
+
+if TYPE_CHECKING:
+    from .feed import _FeedsBus
+
+log = get_logger(name=__name__)
+
+# `datad`-actor-always-enabled mods: the data-side successor
+# to the old `piker.brokers._daemon._data_mods` set.
+# NOTE: keeping this list as small as possible is part of
+# our caps-sec model and should be treated with utmost care!
+_datad_service_mods: list[str] = [
+    'piker.brokers.core',
+    'piker.brokers.data',
+    'piker.data',
+    'piker.data.feed',
+    'piker.data._sampling',
+    'piker.data._daemon',
+]
+
+
+@tractor.context
+async def _setup_persistent_datad(
+    ctx: tractor.Context,
+    brokername: str,
+    loglevel: str|None = None,
+
+) -> None:
+    '''
+    Allocate an actor-wide service nursery in this
+    `datad.<brokername>` actor such that data-feed tasks
+    (shm writer-samplers, history backfillers, symbology
+    loaders) can be run in the background persistently by
+    the provider backend as needed.
+
+    '''
+    # NOTE: we only need to setup logging once (and only)
+    # here since all hosted daemon tasks will reference
+    # this same log instance's (actor local) state and thus
+    # don't require any further (level) configuration on
+    # their own B)
+    actor: tractor.Actor = tractor.current_actor()
+    tll: str = actor.loglevel
+    log = get_console_log(
+        level=loglevel or tll,
+        name=f'{_util.subsys}.{brokername}',
+        with_tractor_log=bool(tll),
+    )
+    assert log.name == _util.subsys
+
+    # XXX: ALSO enable console logging for the provider
+    # backend's mod subtree (eg. `piker.brokers.ib.*`)
+    # since (pre the datad|brokerd-split) that was done
+    # by `brokerd`'s fixture; without this all backend
+    # records here fall through to the stdlib's bare
+    # (non-colorized) `logging.lastResort` handler!
+    get_console_log(
+        level=loglevel or tll,
+        name=f'piker.brokers.{brokername}',
+        with_tractor_log=bool(tll),
+    )
+
+    from piker.data import feed
+    assert not feed._bus
+
+    # allocate a nursery to the bus for spawning background
+    # tasks which service client IPC requests, normally
+    # `tractor.Context` connections to explicitly required
+    # `datad` endpoints such as:
+    # - `stream_quotes()`,
+    # - `manage_history()`,
+    # - `allocate_persistent_feed()`,
+    # - `open_symbol_search()`
+    # NOTE: see ep invocation details inside `.data.feed`.
+    async with (
+        trio.open_nursery() as service_nursery
+    ):
+        bus: _FeedsBus = feed.get_feed_bus(
+            brokername,
+            service_nursery,
+        )
+        assert bus is feed._bus
+
+        # unblock caller
+        await ctx.started()
+
+        # we pin this task to keep the feeds manager active
+        # until the parent actor decides to tear it down
+        await trio.sleep_forever()
+
+
+def datad_init(
+    brokername: str,
+    loglevel: str|None = None,
+
+    **start_actor_kwargs,
+
+) -> tuple[
+    ModuleType,
+    dict,
+    AsyncContextManager,
+]:
+    '''
+    Given an input broker name, load all named arguments
+    which can be passed for daemon endpoint + context spawn
+    as required in every `datad.<brokername>` (actor)
+    service.
+
+    This includes:
+    - load the appropriate <brokername>.py pkg module,
+    - reads any declared `_datad_mods: list[str]` (falling
+      back to the full `__enable_modules__` set for
+      not-yet-split backends) which will be passed to
+      `tractor.ActorNursery.start_actor(enable_modules=)`
+      at actor start time,
+    - deliver a reference to the daemon lifetime fixture,
+      which for now is always the
+      `_setup_persistent_datad()` context defined above.
+
+    '''
+    from piker.brokers import get_brokermod
+    from .validate import get_eps
+    brokermod = get_brokermod(brokername)
+    modpath: str = brokermod.__name__
+
+    # warn (but don't bail) when the backend is missing
+    # some/all of the `datad` ep contract defined by
+    # `piker.data.validate._eps`.
+    datad_eps: dict = get_eps(brokermod, 'datad')
+    if not datad_eps:
+        log.warning(
+            f'Backend {brokername!r} offers NO `datad` '
+            f'(data-feed) eps!?\n'
+            f'Most feed/chart functionality will be '
+            f'broken for this provider..\n'
+        )
+
+    start_actor_kwargs['name'] = f'datad.{brokername}'
+
+    # XXX CRITICAL: include any backend-declared spawn
+    # kwargs, eg. `{'infect_asyncio': True}` required by
+    # `ib`'s embedded `asyncio`-mode `ib_async` usage!
+    start_actor_kwargs.update(
+        getattr(
+            brokermod,
+            '_spawn_kwargs',
+            {},
+        )
+    )
+
+    # lookup actor-enabled modules declared by the backend
+    # offering the `datad` endpoint(s).
+    enabled: list[str]
+    enabled = start_actor_kwargs['enable_modules'] = [
+        __name__,  # so that eps from THIS mod can be invoked
+        modpath,
+    ]
+    for submodname in getattr(
+        brokermod,
+        '_datad_mods',
+        # fallback for (flat, less mature) backends which
+        # don't yet declare a daemon-kind mod split.
+        getattr(
+            brokermod,
+            '__enable_modules__',
+            [],
+        ),
+    ):
+        subpath: str = f'{modpath}.{submodname}'
+        enabled.append(subpath)
+
+    return (
+        brokermod,
+        start_actor_kwargs,  # to `ActorNursery.start_actor()`
+
+        # XXX see impl above; contains all (actor global)
+        # setup/teardown expected in all `datad` actor
+        # instances.
+        _setup_persistent_datad,
+    )
+
+
+async def spawn_datad(
+    brokername: str,
+    loglevel: str|None = None,
+
+    **tractor_kwargs,
+
+) -> bool:
+
+    log.info(
+        f'Spawning data-daemon,\n'
+        f'backend: {brokername!r}'
+    )
+
+    (
+        brokermod,
+        tractor_kwargs,
+        daemon_fixture_ep,
+    ) = datad_init(
+        brokername,
+        loglevel,
+        **tractor_kwargs,
+    )
+
+    # ask `pikerd` to spawn a new sub-actor and manage it
+    # under its actor nursery
+    from piker.service import Services
+
+    dname: str = tractor_kwargs.pop('name')  # f'datad.{brokername}'
+    enable_mods: list[str] = list(dict.fromkeys(
+        _datad_service_mods
+        +
+        tractor_kwargs.pop('enable_modules')
+    ))
+    portal = await Services.actor_n.start_actor(
+        dname,
+        enable_modules=enable_mods,
+        debug_mode=Services.debug_mode,
+        **tractor_kwargs,
+    )
+
+    # NOTE: the service mngr expects an already spawned
+    # actor + its portal ref in order to do non-blocking
+    # setup of the `datad` service nursery.
+    await Services.start_service_task(
+        dname,
+        portal,
+
+        # signature of target root-task endpoint
+        daemon_fixture_ep,
+        brokername=brokername,
+        loglevel=loglevel,
+    )
+    return True
+
+
+@acm
+async def maybe_spawn_datad(
+    brokername: str,
+    loglevel: str|None = None,
+
+    **pikerd_kwargs,
+
+) -> tractor.Portal:
+    '''
+    Helper to spawn a datad service *from* a client who
+    wishes to use the sub-actor-daemon but is fine with
+    re-using any existing and contactable `datad`.
+
+    Mas o menos, acts as a cached-actor-getter factory.
+
+    '''
+    from piker.service import maybe_spawn_daemon
+
+    async with maybe_spawn_daemon(
+        service_name=f'datad.{brokername}',
+        service_task_target=spawn_datad,
+        spawn_args={
+            'brokername': brokername,
+        },
+        loglevel=loglevel,
+
+        **pikerd_kwargs,
+
+    ) as portal:
+        yield portal

piker/data/_web_bs.py

@@ -168,7 +168,7 @@ async def _reconnect_forever(
     nobsws: NoBsWs,
     reset_after: int,  # msg recv timeout before reset attempt
 
-    fixture: AsyncContextManager | None = None,
+    fixture: AsyncContextManager|None = None,
     task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
 
 ) -> None:
@@ -185,7 +185,7 @@ async def _reconnect_forever(
     async def proxy_msgs(
         ws: WebSocketConnection,
         rent_cs: trio.CancelScope,  # parent cancel scope
-    ):
+    ) -> None:
         '''
         Receive (under `timeout` deadline) all msgs from from underlying
         websocket and relay them to (calling) parent task via ``trio``
@@ -206,7 +206,7 @@ async def _reconnect_forever(
                 except nobsws.recon_errors:
                     log.exception(
                         f'{src_mod}\n'
-                        f'{url} connection bail with:'
+                        f'{url!r} connection failed\n'
                     )
                     with trio.CancelScope(shield=True):
                         await trio.sleep(0.5)
@@ -269,7 +269,7 @@ async def _reconnect_forever(
                 nobsws._ws = ws
                 log.info(
                     f'{src_mod}\n'
-                    f'Connection success: {url}'
+                    f'Connection success: {url!r}'
                 )
 
                 # begin relay loop to forward msgs
@@ -341,7 +341,7 @@ async def _reconnect_forever(
 async def open_autorecon_ws(
     url: str,
 
-    fixture: AsyncContextManager | None = None,
+    fixture: AsyncContextManager|None = None,
 
     # time in sec between msgs received before
     # we presume connection might need a reset.
@@ -361,7 +361,7 @@ async def open_autorecon_ws(
     and restarts the full http(s) handshake on catches of certain
     connetivity errors, or some user defined recv timeout.
 
-    You can provide a ``fixture`` async-context-manager which will be
+    You can provide a `fixture` async-context-manager which will be
     entered/exitted around each connection reset; eg. for
     (re)requesting subscriptions without requiring streaming setup
     code to rerun.
@@ -402,7 +402,8 @@ async def open_autorecon_ws(
     except NoBsWs.recon_errors as con_err:
         log.warning(
             f'Entire ws-channel disconnect due to,\n'
-            f'con_err: {con_err!r}\n'
+            f'\n'
+            f'{con_err!r}\n'
         )
 
 
@@ -424,7 +425,7 @@ class JSONRPCResult(Struct):
 async def open_jsonrpc_session(
     url: str,
     start_id: int = 0,
-    response_type: type = JSONRPCResult,
+    response_type: Type[Struct] = JSONRPCResult,
     msg_recv_timeout: float = float('inf'),
     # ^NOTE, since only `deribit` is using this jsonrpc stuff atm
     # and options mkts are generally "slow moving"..
@@ -435,7 +436,10 @@ async def open_jsonrpc_session(
     # broken and never restored with wtv init sequence is required to
     # re-establish a working req-resp session.
 
-) -> Callable[[str, dict], dict]:
+) -> Callable[
+    [str, dict],
+    dict,
+]:
     '''
     Init a json-RPC-over-websocket connection to the provided `url`.
 
@@ -531,14 +535,18 @@ async def open_jsonrpc_session(
                         'id': mid,
                     } if not rpc_results.get(mid):
                         log.warning(
-                            f'Unexpected ws msg: {json.dumps(msg, indent=4)}'
+                            f'Unexpected ws msg?\n'
+                            f'{json.dumps(msg, indent=4)}'
                         )
 
                     case {
                         'method': _,
                         'params': _,
                     }:
-                        log.debug(f'Recieved\n{msg}')
+                        log.debug(
+                            f'Recieved\n'
+                            f'{msg!r}'
+                        )
 
                     case {
                         'error': error
@@ -554,12 +562,15 @@ async def open_jsonrpc_session(
                         result['event'].set()
                         log.error(
                             f'JSONRPC request failed\n'
-                            f'req: {req_msg}\n'
-                            f'resp: {error}\n'
+                            f'req: {req_msg!r}\n'
+                            f'resp: {error!r}\n'
                         )
 
                     case _:
-                        log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
+                        log.warning(
+                            f'Unhandled JSON-RPC msg!?\n'
+                            f'{msg!r}'
+                        )
 
         tn.start_soon(recv_task)
         yield json_rpc

piker/data/feed.py

@@ -17,7 +17,7 @@
 '''
 Data feed apis and infra.
 
-This module is enabled for ``brokerd`` daemons and includes mostly
+This module is enabled for ``datad`` daemons and includes mostly
 endpoints and middleware to support our real-time, provider agnostic,
 live market quotes layer. Historical data loading and processing is also
 initiated in parts of the feed bus startup but business logic and
@@ -54,8 +54,11 @@ from piker.accounting import (
 )
 from piker.types import Struct
 from piker.brokers import get_brokermod
-from piker.service import (
-    maybe_spawn_brokerd,
+# NOTE: must be a "relative-direct" import (NOT via
+# `piker.service`) to avoid a partial-init cycle when this
+# mod is loaded as part of `piker.service.__init__`.
+from ._daemon import (
+    maybe_spawn_datad,
 )
 from piker.calc import humanize
 from ._util import (
@@ -77,7 +80,7 @@ from ._sampling import (
 
 if TYPE_CHECKING:
     from .flows import Flume
-    from tractor._addr import Address
+    from tractor.discovery._addr import Address
     from tractor.msg.types import Aid
 
 
@@ -110,7 +113,7 @@ class _FeedsBus(Struct):
     '''
     Data feeds broadcaster and persistence management.
 
-    This is a brokerd side api used to manager persistent real-time
+    This is a datad side api used to manager persistent real-time
     streams that can be allocated and left alive indefinitely. A bus is
     associated one-to-one with a particular broker backend where the
     "bus" refers so a multi-symbol bus where quotes are interleaved in
@@ -249,7 +252,7 @@ async def allocate_persistent_feed(
     '''
     Create and maintain a "feed bus" which allocates tasks for real-time
     streaming and optional historical data storage per broker/data provider
-    backend; this normally task runs *in* a `brokerd` actor.
+    backend; this normally task runs *in* a `datad` actor.
 
     If none exists, this allocates a ``_FeedsBus`` which manages the
     lifetimes of streaming tasks created for each requested symbol.
@@ -318,8 +321,8 @@ async def allocate_persistent_feed(
     #   at max capacity.
     # - the same ideas ^ but when a local core is maxxed out (like how
     #   binance does often with hft XD
-    # - if a brokerd is non-local then we can't just allocate a mem
-    #   channel here and have the brokerd write it, we instead need
+    # - if a datad is non-local then we can't just allocate a mem
+    #   channel here and have the datad write it, we instead need
     #   a small streaming machine around the remote feed which can then
     #   do the normal work of sampling and writing shm buffers
     #   (depending on if we want sampling done on the far end or not?)
@@ -499,7 +502,7 @@ async def open_feed_bus(
     # (after we also group them in a nice `/dev/shm/piker/` subdir).
     # ensure we are who we think we are
     servicename = tractor.current_actor().name
-    assert 'brokerd' in servicename
+    assert 'datad' in servicename
     assert brokername in servicename
 
     bus: _FeedsBus = get_feed_bus(brokername)
@@ -509,12 +512,12 @@ async def open_feed_bus(
     for symbol in symbols:
 
         # if no cached feed for this symbol has been created for this
-        # brokerd yet, start persistent stream and shm writer task in
+        # datad yet, start persistent stream and shm writer task in
         # service nursery
         flume = bus.feeds.get(symbol)
         if flume is None:
             # allocate a new actor-local stream bus which
-            # will persist for this `brokerd`'s service lifetime.
+            # will persist for this `datad`'s service lifetime.
             async with bus.task_lock:
                 await bus.nursery.start(
                     partial(
@@ -721,7 +724,7 @@ class Feed(Struct):
             mods = {name: self.mods[name] for name in brokers}
 
         if len(mods) == 1:
-            # just pass the brokerd stream directly if only one provider
+            # just pass the datad stream directly if only one provider
             # was detected.
             stream = self.streams[list(brokers)[0]]
             async with stream.subscribe() as bstream:
@@ -763,7 +766,7 @@ class Feed(Struct):
 
 
 @acm
-async def install_brokerd_search(
+async def install_datad_search(
 
     portal: tractor.Portal,
     brokermod: ModuleType,
@@ -812,7 +815,7 @@ async def maybe_open_feed(
     ReceiveChannel[dict[str, Any]],
 ):
     '''
-    Maybe open a data to a ``brokerd`` daemon only if there is no
+    Maybe open a data feed to a ``datad`` daemon only if there is no
     local one for the broker-symbol pair, if one is cached use it wrapped
     in a tractor broadcast receiver.
 
@@ -885,14 +888,14 @@ async def open_feed(
         providers.setdefault(mod, []).append(bs_fqme)
         feed.mods[mod.name] = mod
 
-    # one actor per brokerd for now
-    brokerd_ctxs = []
+    # one actor per datad for now
+    datad_ctxs = []
     for brokermod, bfqmes in providers.items():
 
-        # if no `brokerd` for this backend exists yet we spawn
+        # if no `datad` for this backend exists yet we spawn
         # a daemon actor for it.
-        brokerd_ctxs.append(
-            maybe_spawn_brokerd(
+        datad_ctxs.append(
+            maybe_spawn_datad(
                 brokermod.name,
                 loglevel=loglevel
             )
@@ -900,7 +903,7 @@ async def open_feed(
 
     portals: tuple[tractor.Portal]
     async with trionics.gather_contexts(
-        brokerd_ctxs,
+        datad_ctxs,
     ) as portals:
 
         bus_ctxs: list[AsyncContextManager] = []
@@ -937,9 +940,9 @@ async def open_feed(
                     tick_throttle=tick_throttle,
 
                     # XXX: super important to avoid
-                    # the brokerd from some other
+                    # the datad from some other
                     # backend overruning the task here
-                    # bc some other brokerd took longer
+                    # bc some other datad took longer
                     # to startup before we hit the `.open_stream()`
                     # loop below XD .. really we should try to do each
                     # of these stream open sequences sequentially per
@@ -973,9 +976,6 @@ async def open_feed(
                     # assert flume.mkt.fqme == fqme
                     feed.flumes[fqme] = flume
 
-                    # TODO: do we need this?
-                    flume.feed = feed
-
                     # attach and cache shm handles
                     rt_shm = flume.rt_shm
                     assert rt_shm
@@ -1011,7 +1011,7 @@ async def open_feed(
                     assert stream
                     feed.streams[brokermod.name] = stream
 
-                    # apply `brokerd`-common stream to each flume
+                    # apply `datad`-common stream to each flume
                     # tracking a live market feed from that provider.
                     for fqme, flume in feed.flumes.items():
                         if brokermod.name == flume.mkt.broker:

piker/data/flows.py

@@ -22,9 +22,6 @@ real-time data processing data-structures.
 
 """
 from __future__ import annotations
-from typing import (
-    TYPE_CHECKING,
-)
 
 import tractor
 import pendulum
@@ -38,9 +35,6 @@ from tractor.ipc._shm import (
 )
 from piker.accounting import MktPair
 
-if TYPE_CHECKING:
-    from piker.data.feed import Feed
-
 
 class Flume(Struct):
     '''
@@ -80,10 +74,6 @@ class Flume(Struct):
     izero_rt: int = 0
     throttle_rate: int | None = None
 
-    # TODO: do we need this really if we can pull the `Portal` from
-    # ``tractor``'s internals?
-    feed: Feed|None = None
-
     @property
     def rt_shm(self) -> ShmArray:
 
@@ -156,7 +146,6 @@ class Flume(Struct):
         # will get instead some kind of msg-compat version
         # that it can load.
         msg.pop('stream')
-        msg.pop('feed')
         msg.pop('_rt_shm')
         msg.pop('_hist_shm')
 

piker/data/validate.py

@@ -28,6 +28,7 @@ from typing import (
 )
 
 from msgspec import field
+from tractor._exceptions import reg_err_types
 
 from piker.types import Struct
 from piker.accounting import (
@@ -43,6 +44,8 @@ class FeedInitializationError(ValueError):
 
     '''
 
+reg_err_types([FeedInitializationError])
+
 
 class FeedInit(Struct, frozen=True):
     '''
@@ -88,6 +91,24 @@ _eps: dict[str, list[str]] = {
 }
 
 
+def get_eps(
+    mod: ModuleType,
+    kind: str,  # 'middleware' | 'datad' | 'brokerd'
+
+) -> dict[str, Callable]:
+    '''
+    Return the daemon-kind's ep funcs defined by the backend
+    `mod`, keyed by ep name; any eps from `_eps[kind]` not
+    implemented by the backend are excluded.
+
+    '''
+    return {
+        name: ep
+        for name in _eps[kind]
+        if (ep := getattr(mod, name, None))
+    }
+
+
 def validate_backend(
     mod: ModuleType,
     syms: list[str],

piker/service/__init__.py

@@ -56,3 +56,7 @@ from ..brokers._daemon import (
     spawn_brokerd as spawn_brokerd,
     maybe_spawn_brokerd as maybe_spawn_brokerd,
 )
+from ..data._daemon import (
+    spawn_datad as spawn_datad,
+    maybe_spawn_datad as maybe_spawn_datad,
+)

piker/service/_actor_runtime.py

@@ -59,6 +59,7 @@ def get_runtime_vars() -> dict[str, Any]:
 async def open_piker_runtime(
     name: str,
     registry_addrs: list[tuple[str, int]] = [],
+    tpt_bind_addrs: list|None = None,
 
     enable_modules: list[str] = [],
     loglevel: str|None = None,
@@ -91,7 +92,7 @@ async def open_piker_runtime(
     try:
         actor = tractor.current_actor()
     except tractor._exceptions.NoRuntime:
-        tractor._state._runtime_vars[
+        tractor.runtime._state._runtime_vars[
             'piker_vars'
         ] = tractor_runtime_overrides
 
@@ -112,6 +113,7 @@ async def open_piker_runtime(
 
                 # passed through to `open_root_actor`
                 registry_addrs=registry_addrs,
+                tpt_bind_addrs=tpt_bind_addrs,
                 name=name,
                 start_method=start_method,
                 loglevel=loglevel,
@@ -155,6 +157,7 @@ _root_modules: list[str] = [
     __name__,
     'piker.service._daemon',
     'piker.brokers._daemon',
+    'piker.data._daemon',
 
     'piker.clearing._ems',
     'piker.clearing._client',
@@ -166,6 +169,7 @@ _root_modules: list[str] = [
 @acm
 async def open_pikerd(
     registry_addrs: list[tuple[str, int]],
+    tpt_bind_addrs: list|None = None,
     loglevel: str|None = None,
 
     # XXX: you should pretty much never want debug mode
@@ -198,6 +202,7 @@ async def open_pikerd(
             loglevel=loglevel,
             debug_mode=debug_mode,
             registry_addrs=registry_addrs,
+            tpt_bind_addrs=tpt_bind_addrs,
 
             **kwargs,
 
@@ -210,7 +215,16 @@ async def open_pikerd(
         trio.open_nursery() as service_tn,
     ):
         for addr in reg_addrs:
-            if addr not in root_actor.accept_addrs:
+            # normalize to a wrapped `tractor` addr-type;
+            # entries may be raw `tuple`s when passed in
+            # from (test) client code.
+            wladdr = tractor.discovery._addr.wrap_address(
+                addr,
+            )
+            uaddr: tuple = wladdr.unwrap()
+            if (
+                uaddr not in root_actor.accept_addrs
+            ):
                 raise RuntimeError(
                     f'`pikerd` failed to bind on {addr}!\n'
                     'Maybe you have another daemon already running?'
@@ -264,7 +278,7 @@ async def maybe_open_pikerd(
     **kwargs,
 
 ) -> (
-    tractor._portal.Portal
+    tractor.Portal
     |ClassVar[Services]
 ):
     '''

piker/service/_ahab.py

@@ -49,6 +49,7 @@ from requests.exceptions import (
     ReadTimeout,
 )
 
+from tractor._exceptions import reg_err_types
 from piker.log import (
     get_console_log,
     get_logger,
@@ -66,6 +67,11 @@ class DockerNotStarted(Exception):
 class ApplicationLogError(Exception):
     'App in container reported an error in logs'
 
+reg_err_types([
+    DockerNotStarted,
+    ApplicationLogError,
+])
+
 
 @acm
 async def open_docker(

piker/service/_daemon.py

@@ -79,10 +79,17 @@ async def maybe_spawn_daemon(
     lock = Services.locks[service_name]
     await lock.acquire()
 
+    if not pikerd_kwargs:
+        # XXX NOTE, pin to apprope `tractor` branch!
+        rtvs: dict = tractor.get_runtime_vars()
+        registry_addrs: list[tuple] = list(
+            map(tuple, rtvs['_registry_addrs'])
+        )
+
     try:
         async with find_service(
             service_name,
-            registry_addrs=[('127.0.0.1', 6116)],
+            registry_addrs=registry_addrs,
         ) as portal:
             if portal is not None:
                 lock.release()
@@ -99,6 +106,7 @@ async def maybe_spawn_daemon(
         # process tree
         async with maybe_open_pikerd(
             loglevel=loglevel,
+            registry_addrs=registry_addrs,
             **pikerd_kwargs,
 
         ) as pikerd_portal:
@@ -142,7 +150,65 @@ async def maybe_spawn_daemon(
             async with tractor.wait_for_actor(service_name) as portal:
                 lock.release()
                 yield portal
-                await portal.cancel_actor()
+                # --- ---- ---
+                # XXX NOTE XXX
+                # --- ---- ---
+                # DO NOT PUT A `portal.cancel_actor()` here (as was prior)!
+                #
+                # Doing so will cause an "out-of-band" ctxc
+                # (`tractor.ContextCancelled`) to be raised inside the
+                # `ServiceMngr.open_context_in_task()`'s call to
+                # `ctx.wait_for_result()` AND the internal self-ctxc
+                # "graceful capture" WILL NOT CATCH IT!
+                #
+                # This can cause certain types of operations to raise
+                # that ctxc BEFORE THEY `return`, resulting in
+                # a "false-negative" ctxc being raised when really
+                # nothing actually failed, other then our semantic
+                # "failure" to suppress an expected, graceful,
+                # self-cancel scenario..
+                #
+                # bUt wHy duZ It WorK lIKe dis..
+                # ------------------------------
+                # from the perspective of the `tractor.Context` this
+                # cancel request was conducted "out of band" since
+                # `Context.cancel()` was never called and thus the
+                # `._cancel_called: bool` was never set. Despite the
+                # remote `.canceller` being set to `pikerd` (i.e. the
+                # same `Actor.uid` of the raising service-mngr task) the
+                # service-task's ctx itself was never marked as having
+                # requested cancellation and thus still raises the ctxc
+                # bc it was unaware of any such request.
+                #
+                # How to make grokin these cases easier tho?
+                # ------------------------------------------
+                # Because `Portal.cancel_actor()` was called it requests
+                # "full-`Actor`-runtime-cancellation" of it's peer
+                # process which IS NOT THE SAME as a single inter-actor
+                # RPC task cancelling its local context with a remote
+                # peer `Task` in that same peer process.
+                #
+                # ?TODO? It might be better if we do one (or all) of the
+                # following:
+                #
+                # -[ ] at least set a special message for the
+                #    `ContextCancelled` when raised locally by the
+                #    unaware ctx task such that we check for the
+                #    `.canceller` being *our `Actor`* and in the case
+                #    where `Context._cancel_called == False` we specially
+                #    note that this is likely an "out-of-band"
+                #    runtime-cancel request triggered by some call to
+                #    `Portal.cancel_actor()`, possibly even reporting the
+                #    exact LOC of that caller by tracking it inside our
+                #    portal-type?
+                # -[ ] possibly add another field `ContextCancelled` like
+                #    maybe a,
+                #    `.request_type: Literal['os', 'proc', 'actor',
+                #    'ctx']` type thing which would allow immediately
+                #    being able to tell what kind of cancellation caused
+                #    the unexpected ctxc?
+                # -[ ] REMOVE THIS COMMENT, once we've settled on how to
+                #     better augment `tractor` to be more explicit on this!
 
     except BaseException as _err:
         err = _err
@@ -152,11 +218,13 @@ async def maybe_spawn_daemon(
             lock.statistics().owner is current_task()
         ):
             log.exception(
-                f'Releasing stale lock after crash..?'
+                f'Releasing stale lock after crash..?\n'
+                f'\n'
                 f'{err!r}\n'
             )
             lock.release()
-            raise err
+
+        raise
 
 
 async def spawn_emsd(

piker/service/_mngr.py

@@ -48,7 +48,7 @@ log = get_logger(name=__name__)
 #   new actors and supervises them to completion?
 class Services:
 
-    actor_n: tractor._supervise.ActorNursery
+    actor_n: tractor.ActorNursery
     service_n: trio.Nursery
     debug_mode: bool  # tractor sub-actor debug mode flag
     service_tasks: dict[

piker/service/_registry.py

@@ -225,10 +225,13 @@ async def check_for_service(
 
     '''
     async with (
-        open_registry(ensure_exists=False) as reg_addr,
+        open_registry(
+            addrs=Registry.addrs,
+            ensure_exists=False,
+        ) as reg_addrs,
         tractor.query_actor(
             service_name,
-            arbiter_sockaddr=reg_addr,
-        ) as sockaddr,
+            regaddr=reg_addrs[0],
+        ) as (sockaddr, _),
     ):
         return sockaddr

piker/service/marketstore.py

@@ -42,6 +42,7 @@ from msgspec.msgpack import (
 # import pyqtgraph as pg
 import numpy as np
 import tractor
+from tractor._exceptions import reg_err_types
 from trio_websocket import open_websocket_url
 from anyio_marketstore import (  # noqa
     open_marketstore_client,
@@ -382,6 +383,8 @@ def quote_to_marketstore_structarray(
 class MarketStoreError(Exception):
     "Generic marketstore client error"
 
+reg_err_types([MarketStoreError])
+
 
 # def err_on_resp(response: dict) -> None:
 #     """Raise any errors found in responses from client request.

piker/storage/__init__.py

@@ -42,15 +42,17 @@ from typing import (
 )
 
 import numpy as np
+from tractor._exceptions import reg_err_types
 
-from .. import config
-from ..service import (
-    check_for_service,
-)
-from ..log import (
+from piker import config
+from piker.log import (
     get_logger,
     get_console_log,
 )
+from piker.service import (
+    check_for_service,
+)
+
 subsys: str = 'piker.storage'
 
 log = get_logger(subsys)
@@ -151,6 +153,12 @@ class StorageConnectionError(ConnectionError):
 
     '''
 
+reg_err_types([
+    TimeseriesNotFound,
+    StorageConnectionError,
+])
+
+
 def get_storagemod(
     name: str,
 

piker/storage/cli.py

@@ -292,6 +292,11 @@ def ldshm(
                             f'Something is wrong with time period for {shm}:\n{times}'
                         )
                 period_s: float = float(max(d1, d2, med))
+                log.info(
+                    f'Processing shm buffer:\n'
+                    f'  file: {shmfile.name}\n'
+                    f'  period: {period_s}s\n'
+                )
 
                 null_segs: tuple = tsp.get_null_segs(
                     frame=shm.array,
@@ -301,7 +306,7 @@ def ldshm(
                 # TODO: call null-seg fixer somehow?
                 if null_segs:
 
-                    if tractor._state.is_debug_mode():
+                    if tractor.runtime._state.is_debug_mode():
                         await tractor.pause()
                 #     async with (
                 #         trio.open_nursery() as tn,

piker/tsp/_anal.py

@@ -276,14 +276,41 @@ def get_null_segs(
         absi_zdiff: np.ndarray = np.diff(absi_zeros)
 
         if zero_t.size < 2:
-            try:
-                breakpoint()
-            except RuntimeError:
-                # XXX, if greenback not active from
-                # piker store ldshm cmd..
-                log.exception(
-                    "Can't debug single-sample null!\n"
+            idx: int = zero_t['index'][0]
+            idx_before: int = idx - 1
+            idx_after: int = idx + 1
+            index = frame['index']
+            before_cond = idx_before <= index
+            after_cond = index <= idx_after
+            bars: np.ndarray = frame[
+                before_cond
+                &
+                after_cond
+            ]
+            time: np.ndarray = bars['time']
+            from pendulum import (
+                from_timestamp,
+                Interval,
             )
+            gap: Interval = (
+                from_timestamp(time[-1])
+                -
+                from_timestamp(time[0])
+            )
+            log.warning(
+                f'Single OHLCV-bar null-segment detected??\n'
+                f'gap -> {gap}\n'
+            )
+
+            # ^^XXX, if you want to debug the above bar-gap^^
+            # try:
+            #     breakpoint()
+            # except RuntimeError:
+            #     # XXX, if greenback not active from
+            #     # piker store ldshm cmd..
+            #     log.exception(
+            #         "Can't debug single-sample null!\n"
+            #     )
 
             return None
 

piker/tsp/_annotate.py

@@ -30,6 +30,11 @@ import tractor
 
 from piker.data._formatters import BGM
 from piker.storage import log
+from piker.toolz.profile import (
+    Profiler,
+    pg_profile_enabled,
+    ms_slower_then,
+)
 from piker.ui._style import get_fonts
 
 if TYPE_CHECKING:
@@ -92,12 +97,22 @@ async def markup_gaps(
     # gap's duration.
     show_txt: bool = False,
 
+    # A/B comparison: render individual arrows alongside batch
+    # for visual comparison
+    show_individual_arrows: bool = False,
+
 ) -> dict[int, dict]:
     '''
     Remote annotate time-gaps in a dt-fielded ts (normally OHLC)
     with rectangles.
 
     '''
+    profiler = Profiler(
+        msg=f'markup_gaps() for {gaps.height} gaps',
+        disabled=False,
+        ms_threshold=0.0,
+    )
+
     # XXX: force chart redraw FIRST to ensure PlotItem coordinate
     # system is properly initialized before we position annotations!
     # Without this, annotations may be misaligned on first creation
@@ -106,6 +121,19 @@ async def markup_gaps(
         fqme=fqme,
         timeframe=timeframe,
     )
+    profiler('first `.redraw()` before annot creation')
+
+    log.info(
+        f'markup_gaps() called:\n'
+        f'  fqme: {fqme}\n'
+        f'  timeframe: {timeframe}s\n'
+        f'  gaps.height: {gaps.height}\n'
+    )
+
+    # collect all annotation specs for batch submission
+    rect_specs: list[dict] = []
+    arrow_specs: list[dict] = []
+    text_specs: list[dict] = []
 
     aids: dict[int] = {}
     for i in range(gaps.height):
@@ -168,7 +196,7 @@ async def markup_gaps(
             prev_r: pl.DataFrame = prev_row_by_i
 
             # debug any missing pre-row
-            if tractor._state.is_debug_mode():
+            if tractor.runtime._state.is_debug_mode():
                 await tractor.pause()
 
         istart: int = prev_r['index'][0]
@@ -217,56 +245,38 @@ async def markup_gaps(
         #     1: 'wine',  # down-gap
         # }[sgn]
 
-        rect_kwargs: dict[str, Any] = dict(
-            fqme=fqme,
-            timeframe=timeframe,
+        # collect rect spec (no fqme/timeframe, added by batch
+        # API)
+        rect_spec: dict[str, Any] = dict(
+            meth='set_view_pos',
             start_pos=lc,
             end_pos=ro,
             color=color,
+            update_label=False,
             start_time=start_time,
             end_time=end_time,
         )
+        rect_specs.append(rect_spec)
 
-        # add up/down rects
-        aid: int|None = await actl.add_rect(**rect_kwargs)
-        if aid is None:
-            log.error(
-                f'Failed to add rect for,\n'
-                f'{rect_kwargs!r}\n'
-                f'\n'
-                f'Skipping to next gap!\n'
-            )
-            continue
-
-        assert aid
-        aids[aid] = rect_kwargs
         direction: str = (
             'down' if down_gap
             else 'up'
         )
-        # TODO! mk this a `msgspec.Struct` which we deserialize
-        # on the server side!
-        # XXX: send timestamp for server-side index lookup
-        # to ensure alignment with current shm state
+
+        # collect arrow spec
         gap_time: float = row['time'][0]
-        arrow_kwargs: dict[str, Any] = dict(
-            fqme=fqme,
-            timeframe=timeframe,
+        arrow_spec: dict[str, Any] = dict(
             x=iend,  # fallback if timestamp lookup fails
             y=cls,
             time=gap_time,  # for server-side index lookup
             color=color,
             alpha=169,
             pointing=direction,
-            # TODO: expose these as params to markup_gaps()?
             headLen=10,
             headWidth=2.222,
             pxMode=True,
         )
-
-        aid: int = await actl.add_arrow(
-            **arrow_kwargs
-        )
+        arrow_specs.append(arrow_spec)
 
         # add duration label to RHS of arrow
         if up_gap:
@@ -278,15 +288,12 @@ async def markup_gaps(
             assert flat
             anchor = (0, 0)  # up from bottom
 
-        # use a slightly smaller font for gap label txt.
+        # collect text spec if enabled
+        if show_txt:
             font, small_font = get_fonts()
             font_size: int = small_font.px_size - 1
-        assert isinstance(font_size, int)
 
-        if show_txt:
-            text_aid: int = await actl.add_text(
-                fqme=fqme,
-                timeframe=timeframe,
+            text_spec: dict[str, Any] = dict(
                 text=gap_label,
                 x=iend + 1,  # fallback if timestamp lookup fails
                 y=cls,
@@ -295,12 +302,46 @@ async def markup_gaps(
                 anchor=anchor,
                 font_size=font_size,
             )
-            aids[text_aid] = {'text': gap_label}
+            text_specs.append(text_spec)
 
-    # tell chart to redraw all its
-    # graphics view layers Bo
+    # submit all annotations in single batch IPC msg
+    log.info(
+        f'Submitting batch annotations:\n'
+        f'  rects: {len(rect_specs)}\n'
+        f'  arrows: {len(arrow_specs)}\n'
+        f'  texts: {len(text_specs)}\n'
+    )
+    profiler('built all annotation specs')
+
+    result: dict[str, list[int]] = await actl.add_batch(
+        fqme=fqme,
+        timeframe=timeframe,
+        rects=rect_specs,
+        arrows=arrow_specs,
+        texts=text_specs,
+        show_individual_arrows=show_individual_arrows,
+    )
+    profiler('batch `.add_batch()` IPC call complete')
+
+    # build aids dict from batch results
+    for aid in result['rects']:
+        aids[aid] = {'type': 'rect'}
+    for aid in result['arrows']:
+        aids[aid] = {'type': 'arrow'}
+    for aid in result['texts']:
+        aids[aid] = {'type': 'text'}
+
+    log.info(
+        f'Batch submission complete: {len(aids)} annotation(s) '
+        f'created'
+    )
+    profiler('built aids result dict')
+
+    # tell chart to redraw all its graphics view layers
     await actl.redraw(
         fqme=fqme,
         timeframe=timeframe,
     )
+    profiler('final `.redraw()` after annot creation')
+
     return aids

piker/tsp/_history.py

@@ -738,12 +738,21 @@ async def start_backfill(
                 # including the dst[/src] source asset token. SO,
                 # 'tsla.nasdaq.ib' over 'tsla/usd.nasdaq.ib' for
                 # historical reasons ONLY.
-                if mkt.dst.atype not in {
+                if (
+                    mkt.dst.atype not in {
                         'crypto',
                         'crypto_currency',
                         'fiat',  # a "forex pair"
                         'perpetual_future',  # stupid "perps" from cex land
-                }:
+                    }
+                    and not (
+                        mkt.src.atype == 'crypto_currency'
+                        and
+                        mkt.dst.atype in {
+                            'future',
+                        }
+                    )
+                ):
                     col_sym_key: str = mkt.get_fqme(
                         delim_char='',
                         without_src=True,

piker/ui/_annotate.py

@@ -24,8 +24,11 @@ from pyqtgraph import (
     Point,
     functions as fn,
     Color,
+    GraphicsObject,
 )
+from pyqtgraph.Qt import internals
 import numpy as np
+import pyqtgraph as pg
 
 from piker.ui.qt import (
     QtCore,
@@ -35,6 +38,10 @@ from piker.ui.qt import (
     QRectF,
     QGraphicsPathItem,
 )
+from piker.ui._style import hcolor
+from piker.log import get_logger
+
+log = get_logger(__name__)
 
 
 def mk_marker_path(
@@ -104,7 +111,7 @@ def mk_marker_path(
 
 class LevelMarker(QGraphicsPathItem):
     '''
-    An arrow marker path graphich which redraws itself
+    An arrow marker path graphic which redraws itself
     to the specified view coordinate level on each paint cycle.
 
     '''
@@ -251,9 +258,9 @@ def qgo_draw_markers(
 
 ) -> float:
     '''
-    Paint markers in ``pg.GraphicsItem`` style by first
-    removing the view transform for the painter, drawing the markers
-    in scene coords, then restoring the view coords.
+    Paint markers in ``pg.GraphicsItem`` style by first removing the
+    view transform for the painter, drawing the markers in scene
+    coords, then restoring the view coords.
 
     '''
     # paint markers in native coordinate system
@@ -295,3 +302,449 @@ def qgo_draw_markers(
 
     p.setTransform(orig_tr)
     return max(sizes)
+
+
+class GapAnnotations(GraphicsObject):
+    '''
+    Batch-rendered gap annotations using Qt's efficient drawing
+    APIs.
+
+    Instead of creating individual `QGraphicsItem` instances per
+    gap (which is very slow for 1000+ gaps), this class stores all
+    gap rectangles and arrows in numpy-backed arrays and renders
+    them in single batch paint calls.
+
+    Performance: ~1000x faster than individual items for large gap
+    counts.
+
+    Based on patterns from:
+    - `pyqtgraph.BarGraphItem` (batch rect rendering)
+    - `pyqtgraph.ScatterPlotItem` (fragment rendering)
+    - `piker.ui._curve.FlowGraphic` (single path pattern)
+
+    '''
+    def __init__(
+        self,
+        gap_specs: list[dict],
+        array: np.ndarray|None = None,
+        color: str = 'dad_blue',
+        alpha: int = 169,
+        arrow_size: float = 10.0,
+        fqme: str|None = None,
+        timeframe: float|None = None,
+    ) -> None:
+        '''
+        gap_specs: list of dicts with keys:
+            - start_pos: (x, y) tuple for left corner of rect
+            - end_pos: (x, y) tuple for right corner of rect
+            - arrow_x: x position for arrow
+            - arrow_y: y position for arrow
+            - pointing: 'up' or 'down' for arrow direction
+            - start_time: (optional) timestamp for repositioning
+            - end_time: (optional) timestamp for repositioning
+
+        array: optional OHLC numpy array for repositioning on
+               backfill updates (when abs-index changes)
+
+        fqme: symbol name for these gaps (for logging/debugging)
+
+        timeframe: period in seconds that these gaps were
+                   detected on (used to skip reposition when
+                   called with wrong timeframe's array)
+
+        '''
+        super().__init__()
+        self._gap_specs = gap_specs
+        self._array = array
+        self._fqme = fqme
+        self._timeframe = timeframe
+        n_gaps = len(gap_specs)
+
+        # shared pen/brush matching original SelectRect/ArrowItem style
+        base_color = pg.mkColor(hcolor(color))
+
+        # rect pen: base color, fully opaque for outline
+        self._rect_pen = pg.mkPen(base_color, width=1)
+
+        # rect brush: base color with alpha=66 (SelectRect default)
+        rect_fill = pg.mkColor(hcolor(color))
+        rect_fill.setAlpha(66)
+        self._rect_brush = pg.functions.mkBrush(rect_fill)
+
+        # arrow pen: same as rects
+        self._arrow_pen = pg.mkPen(base_color, width=1)
+
+        # arrow brush: base color with user-specified alpha (default 169)
+        arrow_fill = pg.mkColor(hcolor(color))
+        arrow_fill.setAlpha(alpha)
+        self._arrow_brush = pg.functions.mkBrush(arrow_fill)
+
+        # allocate rect array using Qt's efficient storage
+        self._rectarray = internals.PrimitiveArray(
+            QtCore.QRectF,
+            4,
+        )
+        self._rectarray.resize(n_gaps)
+        rect_memory = self._rectarray.ndarray()
+
+        # fill rect array from gap specs
+        for (
+            i,
+            spec,
+        ) in enumerate(gap_specs):
+            (
+                start_x,
+                start_y,
+            ) = spec['start_pos']
+            (
+                end_x,
+                end_y,
+            ) = spec['end_pos']
+
+            # QRectF expects (x, y, width, height)
+            rect_memory[i, 0] = start_x
+            rect_memory[i, 1] = min(start_y, end_y)
+            rect_memory[i, 2] = end_x - start_x
+            rect_memory[i, 3] = abs(end_y - start_y)
+
+        # build single QPainterPath for all arrows
+        self._arrow_path = QtGui.QPainterPath()
+        self._arrow_size = arrow_size
+
+        for spec in gap_specs:
+            arrow_x = spec['arrow_x']
+            arrow_y = spec['arrow_y']
+            pointing = spec['pointing']
+
+            # create arrow polygon
+            if pointing == 'down':
+                # arrow points downward
+                arrow_poly = QtGui.QPolygonF([
+                    QPointF(arrow_x, arrow_y),  # tip
+                    QPointF(
+                        arrow_x - arrow_size/2,
+                        arrow_y - arrow_size,
+                    ),  # left
+                    QPointF(
+                        arrow_x + arrow_size/2,
+                        arrow_y - arrow_size,
+                    ),  # right
+                ])
+            else:  # up
+                # arrow points upward
+                arrow_poly = QtGui.QPolygonF([
+                    QPointF(arrow_x, arrow_y),  # tip
+                    QPointF(
+                        arrow_x - arrow_size/2,
+                        arrow_y + arrow_size,
+                    ),  # left
+                    QPointF(
+                        arrow_x + arrow_size/2,
+                        arrow_y + arrow_size,
+                    ),  # right
+                ])
+
+            self._arrow_path.addPolygon(arrow_poly)
+            self._arrow_path.closeSubpath()
+
+        # cache bounding rect
+        self._br: QRectF|None = None
+
+    def boundingRect(self) -> QRectF:
+        '''
+        Compute bounding rect from rect array and arrow path.
+
+        '''
+        if self._br is not None:
+            return self._br
+
+        # get rect bounds
+        rect_memory = self._rectarray.ndarray()
+        if len(rect_memory) == 0:
+            self._br = QRectF()
+            return self._br
+
+        x_min = rect_memory[:, 0].min()
+        y_min = rect_memory[:, 1].min()
+        x_max = (rect_memory[:, 0] + rect_memory[:, 2]).max()
+        y_max = (rect_memory[:, 1] + rect_memory[:, 3]).max()
+
+        # expand for arrow path
+        arrow_br = self._arrow_path.boundingRect()
+        x_min = min(x_min, arrow_br.left())
+        y_min = min(y_min, arrow_br.top())
+        x_max = max(x_max, arrow_br.right())
+        y_max = max(y_max, arrow_br.bottom())
+
+        self._br = QRectF(
+            x_min,
+            y_min,
+            x_max - x_min,
+            y_max - y_min,
+        )
+        return self._br
+
+    def paint(
+        self,
+        p: QtGui.QPainter,
+        opt: QtWidgets.QStyleOptionGraphicsItem,
+        w: QtWidgets.QWidget,
+    ) -> None:
+        '''
+        Batch render all rects and arrows in minimal paint calls.
+
+        '''
+        # draw all rects in single batch call (data coordinates)
+        p.setPen(self._rect_pen)
+        p.setBrush(self._rect_brush)
+        drawargs = self._rectarray.drawargs()
+        p.drawRects(*drawargs)
+
+        # draw arrows in scene/pixel coordinates so they maintain
+        # size regardless of zoom level
+        orig_tr = p.transform()
+        p.resetTransform()
+
+        # rebuild arrow path in scene coordinates
+        arrow_path_scene = QtGui.QPainterPath()
+
+        # arrow geometry matching pg.ArrowItem defaults
+        # headLen=10, headWidth=2.222
+        # headWidth is the half-width (center to edge distance)
+        head_len = self._arrow_size
+        head_width = head_len * 0.2222  # 2.222 at size=10
+
+        for spec in self._gap_specs:
+            if 'arrow_x' not in spec:
+                continue
+
+            arrow_x = spec['arrow_x']
+            arrow_y = spec['arrow_y']
+            pointing = spec['pointing']
+
+            # transform data coords to scene coords
+            scene_pt = orig_tr.map(QPointF(arrow_x, arrow_y))
+            sx = scene_pt.x()
+            sy = scene_pt.y()
+
+            # create arrow polygon in scene/pixel coords
+            # matching pg.ArrowItem geometry but rotated for up/down
+            if pointing == 'down':
+                # tip points downward (negative y direction)
+                arrow_poly = QtGui.QPolygonF([
+                    QPointF(sx, sy),  # tip
+                    QPointF(
+                        sx - head_width,
+                        sy - head_len,
+                    ),  # left base
+                    QPointF(
+                        sx + head_width,
+                        sy - head_len,
+                    ),  # right base
+                ])
+            else:  # up
+                # tip points upward (positive y direction)
+                arrow_poly = QtGui.QPolygonF([
+                    QPointF(sx, sy),  # tip
+                    QPointF(
+                        sx - head_width,
+                        sy + head_len,
+                    ),  # left base
+                    QPointF(
+                        sx + head_width,
+                        sy + head_len,
+                    ),  # right base
+                ])
+
+            arrow_path_scene.addPolygon(arrow_poly)
+            arrow_path_scene.closeSubpath()
+
+        p.setPen(self._arrow_pen)
+        p.setBrush(self._arrow_brush)
+        p.drawPath(arrow_path_scene)
+
+        # restore original transform
+        p.setTransform(orig_tr)
+
+    def reposition(
+        self,
+        array: np.ndarray|None = None,
+        fqme: str|None = None,
+        timeframe: float|None = None,
+    ) -> None:
+        '''
+        Reposition all annotations based on timestamps.
+
+        Used when viz is updated (eg during backfill) and abs-index
+        range changes - we need to lookup new indices from timestamps.
+
+        '''
+        # skip reposition if timeframe doesn't match
+        # (e.g., 1s gaps being repositioned with 60s array)
+        if (
+            timeframe is not None
+            and
+            self._timeframe is not None
+            and
+            timeframe != self._timeframe
+        ):
+            log.debug(
+                f'Skipping reposition for {self._fqme} gaps:\n'
+                f'  gap timeframe: {self._timeframe}s\n'
+                f'  array timeframe: {timeframe}s\n'
+            )
+            return
+
+        if array is None:
+            array = self._array
+
+        if array is None:
+            log.warning(
+                'GapAnnotations.reposition() called but no array '
+                'provided'
+            )
+            return
+
+        # collect all unique timestamps we need to lookup
+        timestamps: set[float] = set()
+        for spec in self._gap_specs:
+            if spec.get('start_time') is not None:
+                timestamps.add(spec['start_time'])
+            if spec.get('end_time') is not None:
+                timestamps.add(spec['end_time'])
+            if spec.get('time') is not None:
+                timestamps.add(spec['time'])
+
+        # vectorized timestamp -> row lookup using binary search
+        time_to_row: dict[float, dict] = {}
+        if timestamps:
+            import numpy as np
+            time_arr = array['time']
+            ts_array = np.array(list(timestamps))
+
+            search_indices = np.searchsorted(
+                time_arr,
+                ts_array,
+            )
+
+            # vectorized bounds check and exact match verification
+            valid_mask = (
+                (search_indices < len(array))
+                & (time_arr[search_indices] == ts_array)
+            )
+
+            valid_indices = search_indices[valid_mask]
+            valid_timestamps = ts_array[valid_mask]
+            matched_rows = array[valid_indices]
+
+            time_to_row = {
+                float(ts): {
+                    'index': float(row['index']),
+                    'open': float(row['open']),
+                    'close': float(row['close']),
+                }
+                for ts, row in zip(
+                    valid_timestamps,
+                    matched_rows,
+                )
+            }
+
+        # rebuild rect array from gap specs with new indices
+        rect_memory = self._rectarray.ndarray()
+
+        for (
+            i,
+            spec,
+        ) in enumerate(self._gap_specs):
+            start_time = spec.get('start_time')
+            end_time = spec.get('end_time')
+
+            if (
+                start_time is None
+                or end_time is None
+            ):
+                continue
+
+            start_row = time_to_row.get(start_time)
+            end_row = time_to_row.get(end_time)
+
+            if (
+                start_row is None
+                or end_row is None
+            ):
+                log.warning(
+                    f'Timestamp lookup failed for gap[{i}] during '
+                    f'reposition:\n'
+                    f'  fqme: {fqme}\n'
+                    f'  timeframe: {timeframe}s\n'
+                    f'  start_time: {start_time}\n'
+                    f'  end_time: {end_time}\n'
+                    f'  array time range: '
+                    f'{array["time"][0]} -> {array["time"][-1]}\n'
+                )
+                continue
+
+            start_idx = start_row['index']
+            end_idx = end_row['index']
+            start_close = start_row['close']
+            end_open = end_row['open']
+
+            from_idx: float = 0.16 - 0.06
+            start_x = start_idx + 1 - from_idx
+            end_x = end_idx + from_idx
+
+            # update rect in array
+            rect_memory[i, 0] = start_x
+            rect_memory[i, 1] = min(start_close, end_open)
+            rect_memory[i, 2] = end_x - start_x
+            rect_memory[i, 3] = abs(end_open - start_close)
+
+        # rebuild arrow path with new indices
+        self._arrow_path.clear()
+
+        for spec in self._gap_specs:
+            time_val = spec.get('time')
+            if time_val is None:
+                continue
+
+            arrow_row = time_to_row.get(time_val)
+            if arrow_row is None:
+                continue
+
+            arrow_x = arrow_row['index']
+            arrow_y = arrow_row['close']
+            pointing = spec['pointing']
+
+            # create arrow polygon
+            if pointing == 'down':
+                arrow_poly = QtGui.QPolygonF([
+                    QPointF(arrow_x, arrow_y),
+                    QPointF(
+                        arrow_x - self._arrow_size/2,
+                        arrow_y - self._arrow_size,
+                    ),
+                    QPointF(
+                        arrow_x + self._arrow_size/2,
+                        arrow_y - self._arrow_size,
+                    ),
+                ])
+            else:  # up
+                arrow_poly = QtGui.QPolygonF([
+                    QPointF(arrow_x, arrow_y),
+                    QPointF(
+                        arrow_x - self._arrow_size/2,
+                        arrow_y + self._arrow_size,
+                    ),
+                    QPointF(
+                        arrow_x + self._arrow_size/2,
+                        arrow_y + self._arrow_size,
+                    ),
+                ])
+
+            self._arrow_path.addPolygon(arrow_poly)
+            self._arrow_path.closeSubpath()
+
+        # invalidate bounding rect cache
+        self._br = None
+        self.prepareGeometryChange()
+        self.update()

piker/ui/_app.py

@@ -32,12 +32,12 @@ from . import _event
 from . import _search
 from ..accounting import unpack_fqme
 from ..data._symcache import open_symcache
-from ..data.feed import install_brokerd_search
+from ..data.feed import install_datad_search
 from ..log import (
     get_logger,
     get_console_log,
 )
-from ..service import maybe_spawn_brokerd
+from ..service import maybe_spawn_datad
 from ._exec import run_qtractor
 
 log = get_logger(__name__)
@@ -50,16 +50,16 @@ async def load_provider_search(
 ) -> None:
 
     name = brokermod.name
-    log.info(f'loading brokerd for {name}..')
+    log.info(f'loading datad for {name}..')
 
     async with (
 
-        maybe_spawn_brokerd(
+        maybe_spawn_datad(
             name,
             loglevel=loglevel
         ) as portal,
 
-        install_brokerd_search(
+        install_datad_search(
             portal,
             brokermod,
         ),
@@ -131,11 +131,11 @@ async def _async_main(
 
     async with (
         tractor.trionics.collapse_eg(),
-        trio.open_nursery() as root_n,
+        trio.open_nursery() as tn,
     ):
         # set root nursery and task stack for spawning other charts/feeds
         # that run cached in the bg
-        godwidget._root_n = root_n
+        godwidget._root_n = tn
 
         # setup search widget and focus main chart view at startup
         # search widget is a singleton alongside the godwidget
@@ -165,7 +165,7 @@ async def _async_main(
             # load other providers into search **after**
             # the chart's select cache
             for brokername, mod in needed_brokermods.items():
-                root_n.start_soon(
+                tn.start_soon(
                     load_provider_search,
                     mod,
                     loglevel,

piker/ui/_axes.py

@@ -20,8 +20,9 @@ Chart axes graphics and behavior.
 """
 from __future__ import annotations
 from functools import lru_cache
-from typing import Callable
 from math import floor
+import platform
+from typing import Callable
 
 import polars as pl
 import pyqtgraph as pg
@@ -42,6 +43,7 @@ from ._style import DpiAwareFont, hcolor, _font
 from ._interaction import ChartView
 from ._dataviz import Viz
 
+_friggin_macos: bool = platform.system() == 'Darwin'
 _axis_pen = pg.mkPen(hcolor('bracket'))
 
 
@@ -75,6 +77,9 @@ class Axis(pg.AxisItem):
         self.pi = plotitem
         self._dpi_font = _font
 
+        # store for later recalculation on zoom
+        self._typical_max_str = typical_max_str
+
         self.setTickFont(_font.font)
         font_size = self._dpi_font.font.pixelSize()
 
@@ -156,6 +161,42 @@ class Axis(pg.AxisItem):
     def size_to_values(self) -> None:
         pass
 
+    def update_fonts(self, font: DpiAwareFont) -> None:
+        '''Update font and recalculate axis sizing after zoom change.'''
+        # IMPORTANT: tell Qt we're about to change geometry
+        self.prepareGeometryChange()
+
+        self._dpi_font = font
+        self.setTickFont(font.font)
+        font_size = font.font.pixelSize()
+
+        # recalculate text offset based on new font size
+        text_offset = None
+        if self.orientation in ('bottom',):
+            text_offset = floor(0.25 * font_size)
+
+        elif self.orientation in ('left', 'right'):
+            text_offset = floor(font_size / 2)
+
+        if text_offset:
+            self.setStyle(tickTextOffset=text_offset)
+
+        # recalculate bounding rect with new font
+        # Note: typical_max_str should be stored from init
+        if not hasattr(self, '_typical_max_str'):
+            self._typical_max_str = '100 000.000  '  # fallback default
+        self.typical_br = font._qfm.boundingRect(self._typical_max_str)
+
+        # Update PyQtGraph's internal text size tracking
+        # This is critical - PyQtGraph uses these internally for auto-expand
+        if self.orientation in ['left', 'right']:
+            self.textWidth = self.typical_br.width()
+        else:
+            self.textHeight = self.typical_br.height()
+
+        # resize axis to fit new font - this triggers PyQtGraph's auto-expand
+        self.size_to_values()
+
     def txt_offsets(self) -> tuple[int, int]:
         return tuple(self.style['tickTextOffset'])
 
@@ -255,9 +296,23 @@ class PriceAxis(Axis):
     ) -> None:
         self._min_tick = size
 
+    if _friggin_macos:
         def size_to_values(self) -> None:
+            # Call PyQtGraph's internal width update mechanism
+            # This respects autoExpandTextSpace and updates min/max constraints
+            self._updateWidth()
+            # tell Qt our preferred size changed so layout recalculates
+            self.updateGeometry()
+            # force parent plot item to recalculate its layout
+            if self.pi and hasattr(self.pi, 'updateGeometry'):
+                self.pi.updateGeometry()
+
+    else:
+        def size_to_values(self) -> None:
+            # XXX, old code!
             self.setWidth(self.typical_br.width())
 
+
     # XXX: drop for now since it just eats up h space
 
     def tickStrings(
@@ -299,7 +354,20 @@ class DynamicDateAxis(Axis):
         1: '%H:%M:%S',
     }
 
+    if _friggin_macos:
         def size_to_values(self) -> None:
+            # Call PyQtGraph's internal height update mechanism
+            # This respects autoExpandTextSpace and updates min/max constraints
+            self._updateHeight()
+            # tell Qt our preferred size changed so layout recalculates
+            self.updateGeometry()
+            # force parent plot item to recalculate its layout
+            if self.pi and hasattr(self.pi, 'updateGeometry'):
+                self.pi.updateGeometry()
+
+    else:
+        def size_to_values(self) -> None:
+            # XXX, old code!
             self.setHeight(self.typical_br.height() + 1)
 
     def _indexes_to_timestrs(

piker/ui/_display.py

@@ -1346,7 +1346,6 @@ async def display_symbol_data(
             fqmes,
             loglevel=loglevel,
             tick_throttle=cycles_per_feed,
-
         ) as feed,
     ):
 
@@ -1461,7 +1460,7 @@ async def display_symbol_data(
 
         async with (
             tractor.trionics.collapse_eg(),
-            trio.open_nursery() as ln,
+            trio.open_nursery() as tn,
         ):
             # if available load volume related built-in display(s)
             vlm_charts: dict[
@@ -1472,7 +1471,7 @@ async def display_symbol_data(
                 flume.has_vlm()
                 and vlm_chart is None
             ):
-                vlm_chart = vlm_charts[fqme] = await ln.start(
+                vlm_chart = vlm_charts[fqme] = await tn.start(
                     open_vlm_displays,
                     rt_linked,
                     flume,
@@ -1480,7 +1479,7 @@ async def display_symbol_data(
 
             # load (user's) FSP set (otherwise known as "indicators")
             # from an input config.
-            ln.start_soon(
+            tn.start_soon(
                 start_fsp_displays,
                 rt_linked,
                 flume,
@@ -1604,11 +1603,11 @@ async def display_symbol_data(
 
             # start update loop task
             dss: dict[str, DisplayState] = {}
-            ln.start_soon(
+            tn.start_soon(
                 partial(
                     graphics_update_loop,
                     dss=dss,
-                    nurse=ln,
+                    nurse=tn,
                     godwidget=godwidget,
                     feed=feed,
                     # min_istream,
@@ -1623,7 +1622,6 @@ async def display_symbol_data(
             order_ctl_fqme: str = fqmes[0]
             mode: OrderMode
             async with (
-
                 open_order_mode(
                     feed,
                     godwidget,

piker/ui/_editors.py

@@ -168,7 +168,7 @@ class ArrowEditor(Struct):
         '''
         uid: str = arrow._uid
         arrows: list[pg.ArrowItem] = self._arrows[uid]
-        log.info(
+        log.debug(
             f'Removing arrow from views\n'
             f'uid: {uid!r}\n'
             f'{arrow!r}\n'
@@ -286,7 +286,9 @@ class LineEditor(Struct):
             for line in lines:
                 line.show_labels()
                 line.hide_markers()
-                log.debug(f'Level active for level: {line.value()}')
+                log.debug(
+                    f'Line active @ level: {line.value()!r}'
+                )
                 # TODO: other flashy things to indicate the order is active
 
         return lines
@@ -329,7 +331,11 @@ class LineEditor(Struct):
                     if line in hovered:
                         hovered.remove(line)
 
-                    log.debug(f'deleting {line} with oid: {uuid}')
+                    log.debug(
+                        f'Deleting level-line\n'
+                        f'line: {line!r}\n'
+                        f'oid: {uuid!r}\n'
+                    )
                     line.delete()
 
                     # make sure the xhair doesn't get left off
@@ -337,7 +343,11 @@ class LineEditor(Struct):
                     cursor.show_xhair()
 
         else:
-            log.warning(f'Could not find line for {line}')
+            log.warning(
+                f'Could not find line for removal ??\n'
+                f'\n'
+                f'{line!r}\n'
+            )
 
         return lines
 
@@ -569,11 +579,11 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
         if update_label:
             self.init_label(view_rect)
 
-        print(
-            'SelectRect modify:\n'
+        log.debug(
+            f'SelectRect modify,\n'
             f'QRectF: {view_rect}\n'
-            f'start_pos: {start_pos}\n'
-            f'end_pos: {end_pos}\n'
+            f'start_pos: {start_pos!r}\n'
+            f'end_pos: {end_pos!r}\n'
         )
         self.show()
 
@@ -640,8 +650,11 @@ class SelectRect(QtWidgets.QGraphicsRectItem):
             dmn=dmn,
         ))
 
-        # print(f'x2, y2: {(x2, y2)}')
-        # print(f'xmn, ymn: {(xmn, ymx)}')
+        # tracing
+        # log.info(
+        #     f'x2, y2: {(x2, y2)}\n'
+        #     f'xmn, ymn: {(xmn, ymx)}\n'
+        # )
 
         label_anchor = Point(
             xmx + 2,

piker/ui/_exec.py

@@ -203,6 +203,9 @@ def run_qtractor(
     if is_windows:
         window.configure_to_desktop()
 
+    # install global keyboard shortcuts for UI zoom
+    window.install_global_zoom_filter()
+
     # actually render to screen
     window.show()
     app.exec_()

piker/ui/_forms.py

@@ -124,6 +124,13 @@ class Edit(QLineEdit):
         self.sizeHint()
         self.update()
 
+    def update_fonts(self, font: DpiAwareFont) -> None:
+        '''Update font and recalculate widget size.'''
+        self.dpi_font = font
+        self.setFont(font.font)
+        # tell Qt our size hint changed so it recalculates layout
+        self.updateGeometry()
+
     def focus(self) -> None:
         self.selectAll()
         self.show()
@@ -241,6 +248,14 @@ class Selection(QComboBox):
         icon_size = round(h * 0.75)
         self.setIconSize(QSize(icon_size, icon_size))
 
+    def update_fonts(self, font: DpiAwareFont) -> None:
+        '''Update font and recalculate widget size.'''
+        self.setFont(font.font)
+        # recalculate heights with new font
+        self.resize()
+        # tell Qt our size hint changed so it recalculates layout
+        self.updateGeometry()
+
     def set_items(
         self,
         keys: list[str],
@@ -431,6 +446,39 @@ class FieldsForm(QWidget):
         self.fields[key] = select
         return select
 
+    def update_fonts(self) -> None:
+        '''Update font sizes after zoom change.'''
+        from ._style import _font, _font_small
+
+        # update stored font size
+        self._font_size = _font_small.px_size - 2
+
+        # update all labels
+        for name, label in self.labels.items():
+            if hasattr(label, 'update_font'):
+                label.update_font(_font.font, self._font_size - 1)
+
+        # update all fields (edits, selects)
+        for key, field in self.fields.items():
+            # first check for our custom update_fonts method (Edit, Selection)
+            if hasattr(field, 'update_fonts'):
+                field.update_fonts(_font)
+            # then handle stylesheet updates for those without custom methods
+            elif hasattr(field, 'setStyleSheet'):
+                # regenerate stylesheet with new font size
+                field.setStyleSheet(
+                    f"""QLineEdit {{
+                        color : {hcolor('gunmetal')};
+                        font-size : {self._font_size}px;
+                    }}
+                    """
+                )
+                field.setFont(_font.font)
+
+            # for Selection widgets that need style updates
+            if hasattr(field, 'set_style'):
+                field.set_style(color='gunmetal', font_size=self._font_size)
+
 
 async def handle_field_input(
 
@@ -633,6 +681,37 @@ class FillStatusBar(QProgressBar):
         self.setRange(0, int(slots))
         self.setValue(value)
 
+    def update_fonts(self, font_size: int) -> None:
+        '''Update font size after zoom change.'''
+        from ._style import _font_small
+
+        self.font_size = font_size
+        # regenerate stylesheet with new font size
+        self.setStyleSheet(
+            f"""
+            QProgressBar {{
+
+                text-align: center;
+
+                font-size : {self.font_size - 2}px;
+
+                background-color: {hcolor('papas_special')};
+                color : {hcolor('papas_special')};
+
+                border: {self.border_px}px solid {hcolor('default_light')};
+                border-radius: 2px;
+            }}
+            QProgressBar::chunk {{
+
+                background-color: {hcolor('default_spotlight')};
+                color: {hcolor('bracket')};
+
+                border-radius: 2px;
+            }}
+            """
+        )
+        self.setFont(_font_small.font)
+
 
 def mk_fill_status_bar(
 

piker/ui/_label.py

@@ -334,3 +334,19 @@ class FormatLabel(QLabel):
         out = self.fmt_str.format(**fields)
         self.setText(out)
         return out
+
+    def update_font(
+        self,
+        font: QtGui.QFont,
+        font_size: int,
+        font_color: str = 'default_lightest',
+    ) -> None:
+        '''Update font after zoom change.'''
+        self.setStyleSheet(
+            f"""QLabel {{
+                color : {hcolor(font_color)};
+                font-size : {font_size}px;
+            }}
+            """
+        )
+        self.setFont(font)

piker/ui/_lines.py

@@ -38,7 +38,6 @@ from piker.ui.qt import (
     QtGui,
     QGraphicsPathItem,
     QStyleOptionGraphicsItem,
-    QGraphicsItem,
     QGraphicsScene,
     QWidget,
     QPointF,

piker/ui/_position.py

@@ -178,6 +178,26 @@ class SettingsPane:
     # encompasing high level namespace
     order_mode: OrderMode | None = None  # typing: ignore # noqa
 
+    def update_fonts(self) -> None:
+        '''Update font sizes after zoom change.'''
+        from ._style import _font_small
+
+        # update form fields
+        if self.form and hasattr(self.form, 'update_fonts'):
+            self.form.update_fonts()
+
+        # update fill status bar
+        if self.fill_bar and hasattr(self.fill_bar, 'update_fonts'):
+            self.fill_bar.update_fonts(_font_small.px_size)
+
+        # update labels with new fonts
+        if self.step_label:
+            self.step_label.setFont(_font_small.font)
+        if self.pnl_label:
+            self.pnl_label.setFont(_font_small.font)
+        if self.limit_label:
+            self.limit_label.setFont(_font_small.font)
+
     def set_accounts(
         self,
         names: list[str],

piker/ui/_remote_ctl.py

@@ -22,6 +22,7 @@ a chart from some other actor.
 from __future__ import annotations
 from contextlib import (
     asynccontextmanager as acm,
+    contextmanager as cm,
     AsyncExitStack,
 )
 from functools import partial
@@ -46,6 +47,7 @@ from piker.log import get_logger
 from piker.types import Struct
 from piker.service import find_service
 from piker.brokers import SymbolNotFound
+from piker.toolz import Profiler
 from piker.ui.qt import (
     QGraphicsItem,
 )
@@ -98,6 +100,8 @@ def rm_annot(
     annot: ArrowEditor|SelectRect|pg.TextItem
 ) -> bool:
     global _editors
+    from piker.ui._annotate import GapAnnotations
+
     match annot:
         case pg.ArrowItem():
             editor = _editors[annot._uid]
@@ -122,9 +126,35 @@ def rm_annot(
                 scene.removeItem(annot)
             return True
 
+        case GapAnnotations():
+            scene = annot.scene()
+            if scene:
+                scene.removeItem(annot)
+            return True
+
     return False
 
 
+@cm
+def no_qt_updates(*items):
+    '''
+    Disable Qt widget/item updates during context to batch
+    render operations and only trigger single repaint on exit.
+
+    Accepts both QWidgets and QGraphicsItems.
+
+    '''
+    for item in items:
+        if hasattr(item, 'setUpdatesEnabled'):
+            item.setUpdatesEnabled(False)
+    try:
+        yield
+    finally:
+        for item in items:
+            if hasattr(item, 'setUpdatesEnabled'):
+                item.setUpdatesEnabled(True)
+
+
 async def serve_rc_annots(
     ipc_key: str,
     annot_req_stream: MsgStream,
@@ -429,6 +459,333 @@ async def serve_rc_annots(
                 aids.add(aid)
                 await annot_req_stream.send(aid)
 
+            case {
+                'cmd': 'batch',
+                'fqme': fqme,
+                'timeframe': timeframe,
+                'rects': list(rect_specs),
+                'arrows': list(arrow_specs),
+                'texts': list(text_specs),
+                'show_individual_arrows': bool(show_individual_arrows),
+            }:
+                # batch submission handler - process multiple
+                # annotations in single IPC round-trip
+                ds: DisplayState = _dss[fqme]
+                try:
+                    chart: ChartPlotWidget = {
+                        60: ds.hist_chart,
+                        1: ds.chart,
+                    }[timeframe]
+                except KeyError:
+                    msg: str = (
+                        f'No chart for timeframe={timeframe}s, '
+                        f'skipping batch annotation'
+                    )
+                    log.error(msg)
+                    await annot_req_stream.send({'error': msg})
+                    continue
+
+                cv: ChartView = chart.cv
+                viz: Viz = chart.get_viz(fqme)
+                shm = viz.shm
+                arr = shm.array
+
+                result: dict[str, list[int]] = {
+                    'rects': [],
+                    'arrows': [],
+                    'texts': [],
+                }
+
+                profiler = Profiler(
+                    msg=(
+                        f'Batch annotate {len(rect_specs)} gaps '
+                        f'on {fqme}@{timeframe}s'
+                    ),
+                    disabled=False,
+                    delayed=False,
+                )
+
+                aids_set: set[int] = ctxs[ipc_key][1]
+
+                # build unified gap_specs for GapAnnotations class
+                from piker.ui._annotate import GapAnnotations
+
+                gap_specs: list[dict] = []
+                n_gaps: int = max(
+                    len(rect_specs),
+                    len(arrow_specs),
+                )
+                profiler('setup batch annot creation')
+
+                # collect all unique timestamps for vectorized lookup
+                timestamps: list[float] = []
+                for rect_spec in rect_specs:
+                    if start_time := rect_spec.get('start_time'):
+                        timestamps.append(start_time)
+                    if end_time := rect_spec.get('end_time'):
+                        timestamps.append(end_time)
+                for arrow_spec in arrow_specs:
+                    if time_val := arrow_spec.get('time'):
+                        timestamps.append(time_val)
+
+                profiler('collect `timestamps: list` complet!')
+
+                # build timestamp -> row mapping using binary search
+                # O(m log n) instead of O(n*m) with np.isin
+                time_to_row: dict[float, dict] = {}
+                if timestamps:
+                    import numpy as np
+                    time_arr = arr['time']
+                    ts_array = np.array(timestamps)
+
+                    # binary search for each timestamp in sorted time array
+                    search_indices = np.searchsorted(
+                        time_arr,
+                        ts_array,
+                    )
+
+                    profiler('`np.searchsorted()` complete!')
+
+                    # vectorized bounds check and exact match verification
+                    valid_mask = (
+                        (search_indices < len(arr))
+                        & (time_arr[search_indices] == ts_array)
+                    )
+
+                    # get all valid indices and timestamps
+                    valid_indices = search_indices[valid_mask]
+                    valid_timestamps = ts_array[valid_mask]
+
+                    # use fancy indexing to get all rows at once
+                    matched_rows = arr[valid_indices]
+
+                    # extract fields to plain arrays BEFORE dict building
+                    indices_arr = matched_rows['index'].astype(float)
+                    opens_arr = matched_rows['open'].astype(float)
+                    closes_arr = matched_rows['close'].astype(float)
+
+                    profiler('extracted field arrays')
+
+                    # build dict from plain arrays (much faster)
+                    time_to_row: dict[float, dict] = {
+                        float(ts): {
+                            'index': idx,
+                            'open': opn,
+                            'close': cls,
+                        }
+                        for (
+                            ts,
+                            idx,
+                            opn,
+                            cls,
+                        ) in zip(
+                            valid_timestamps,
+                            indices_arr,
+                            opens_arr,
+                            closes_arr,
+                        )
+                    }
+
+                    profiler('`time_to_row` creation complete!')
+
+                profiler(f'built timestamp lookup for {len(timestamps)} times')
+
+                # build gap_specs from rect+arrow specs
+                for i in range(n_gaps):
+                    gap_spec: dict = {}
+
+                    # get rect spec for this gap
+                    if i < len(rect_specs):
+                        rect_spec: dict = rect_specs[i].copy()
+                        start_time = rect_spec.get('start_time')
+                        end_time = rect_spec.get('end_time')
+
+                        if (
+                            start_time is not None
+                            and end_time is not None
+                        ):
+                            # lookup from pre-built mapping
+                            start_row = time_to_row.get(start_time)
+                            end_row = time_to_row.get(end_time)
+
+                            if (
+                                start_row is None
+                                or end_row is None
+                            ):
+                                log.warning(
+                                    f'Timestamp lookup failed for '
+                                    f'gap[{i}], skipping'
+                                )
+                                continue
+
+                            start_idx = start_row['index']
+                            end_idx = end_row['index']
+                            start_close = start_row['close']
+                            end_open = end_row['open']
+
+                            from_idx: float = 0.16 - 0.06
+                            gap_spec['start_pos'] = (
+                                start_idx + 1 - from_idx,
+                                start_close,
+                            )
+                            gap_spec['end_pos'] = (
+                                end_idx + from_idx,
+                                end_open,
+                            )
+                            gap_spec['start_time'] = start_time
+                            gap_spec['end_time'] = end_time
+                            gap_spec['color'] = rect_spec.get(
+                                'color',
+                                'dad_blue',
+                            )
+
+                    # get arrow spec for this gap
+                    if i < len(arrow_specs):
+                        arrow_spec: dict = arrow_specs[i].copy()
+                        x: float = float(arrow_spec.get('x', 0))
+                        y: float = float(arrow_spec.get('y', 0))
+                        time_val: float|None = arrow_spec.get('time')
+
+                        # timestamp-based index lookup (only for x, NOT y!)
+                        # y is already set to the PREVIOUS bar's close
+                        if time_val is not None:
+                            arrow_row = time_to_row.get(time_val)
+                            if arrow_row is not None:
+                                x = arrow_row['index']
+                                # NOTE: do NOT update y! it's the
+                                # previous bar's close, not current
+                            else:
+                                log.warning(
+                                    f'Arrow timestamp {time_val} not '
+                                    f'found for gap[{i}], using x={x}'
+                                )
+
+                        gap_spec['arrow_x'] = x
+                        gap_spec['arrow_y'] = y
+                        gap_spec['time'] = time_val
+                        gap_spec['pointing'] = arrow_spec.get(
+                            'pointing',
+                            'down',
+                        )
+                        gap_spec['alpha'] = arrow_spec.get('alpha', 169)
+
+                    gap_specs.append(gap_spec)
+
+                profiler(f'built {len(gap_specs)} gap_specs')
+
+                # create single GapAnnotations item for all gaps
+                if gap_specs:
+                    gaps_item = GapAnnotations(
+                        gap_specs=gap_specs,
+                        array=arr,
+                        color=gap_specs[0].get('color', 'dad_blue'),
+                        alpha=gap_specs[0].get('alpha', 169),
+                        arrow_size=10.0,
+                        fqme=fqme,
+                        timeframe=timeframe,
+                    )
+                    chart.plotItem.addItem(gaps_item)
+
+                    # register single item for repositioning
+                    aid: int = id(gaps_item)
+                    annots[aid] = gaps_item
+                    aids_set.add(aid)
+                    result['rects'].append(aid)
+                    profiler(
+                        f'created GapAnnotations item for {len(gap_specs)} '
+                        f'gaps'
+                    )
+
+                    # A/B comparison: optionally create individual arrows
+                    # alongside batch for visual comparison
+                    if show_individual_arrows:
+                        godw = chart.linked.godwidget
+                        arrows: ArrowEditor = ArrowEditor(godw=godw)
+                        for i, spec in enumerate(gap_specs):
+                            if 'arrow_x' not in spec:
+                                continue
+
+                            aid_str: str = str(uuid4())
+                            arrow: pg.ArrowItem = arrows.add(
+                                plot=chart.plotItem,
+                                uid=aid_str,
+                                x=spec['arrow_x'],
+                                y=spec['arrow_y'],
+                                pointing=spec['pointing'],
+                                color='bracket',  # different color
+                                alpha=spec.get('alpha', 169),
+                                headLen=10.0,
+                                headWidth=2.222,
+                                pxMode=True,
+                            )
+                            arrow._abs_x = spec['arrow_x']
+                            arrow._abs_y = spec['arrow_y']
+
+                            annots[aid_str] = arrow
+                            _editors[aid_str] = arrows
+                            aids_set.add(aid_str)
+                            result['arrows'].append(aid_str)
+
+                        profiler(
+                            f'created {len(gap_specs)} individual arrows '
+                            f'for comparison'
+                        )
+
+                # handle text items separately (less common, keep
+                # individual items)
+                n_texts: int = 0
+                for text_spec in text_specs:
+                    kwargs: dict = text_spec.copy()
+                    text: str = kwargs.pop('text')
+                    x: float = float(kwargs.pop('x'))
+                    y: float = float(kwargs.pop('y'))
+                    time_val: float|None = kwargs.pop('time', None)
+
+                    # timestamp-based index lookup
+                    if time_val is not None:
+                        matches = arr[arr['time'] == time_val]
+                        if len(matches) > 0:
+                            x = float(matches[0]['index'])
+                            y = float(matches[0]['close'])
+
+                    color = kwargs.pop('color', 'dad_blue')
+                    anchor = kwargs.pop('anchor', (0, 1))
+                    font_size = kwargs.pop('font_size', None)
+
+                    text_item: pg.TextItem = pg.TextItem(
+                        text,
+                        color=hcolor(color),
+                        anchor=anchor,
+                    )
+
+                    if font_size is None:
+                        from ._style import get_fonts
+                        font, font_small = get_fonts()
+                        font_size = font_small.px_size - 1
+
+                    qfont: QFont = text_item.textItem.font()
+                    qfont.setPixelSize(font_size)
+                    text_item.setFont(qfont)
+
+                    text_item.setPos(float(x), float(y))
+                    chart.plotItem.addItem(text_item)
+
+                    text_item._abs_x = float(x)
+                    text_item._abs_y = float(y)
+
+                    aid: str = str(uuid4())
+                    annots[aid] = text_item
+                    aids_set.add(aid)
+                    result['texts'].append(aid)
+                    n_texts += 1
+
+                profiler(
+                    f'created text annotations: {n_texts} texts'
+                )
+                profiler.finish()
+
+                await annot_req_stream.send(result)
+
             case {
                 'cmd': 'remove',
                 'aid': int(aid)|str(aid),
@@ -471,10 +828,26 @@ async def serve_rc_annots(
                 # XXX: reposition all annotations to ensure they
                 # stay aligned with viz data after reset (eg during
                 # backfill when abs-index range changes)
+                chart: ChartPlotWidget = {
+                    60: ds.hist_chart,
+                    1: ds.chart,
+                }[timeframe]
+                viz: Viz = chart.get_viz(fqme)
+                arr = viz.shm.array
+
                 n_repositioned: int = 0
                 for aid, annot in annots.items():
+                    # GapAnnotations batch items have .reposition()
+                    if hasattr(annot, 'reposition'):
+                        annot.reposition(
+                            array=arr,
+                            fqme=fqme,
+                            timeframe=timeframe,
+                        )
+                        n_repositioned += 1
+
                     # arrows and text items use abs x,y coords
-                    if (
+                    elif (
                         hasattr(annot, '_abs_x')
                         and
                         hasattr(annot, '_abs_y')
@@ -539,12 +912,21 @@ async def remote_annotate(
         finally:
             # ensure all annots for this connection are deleted
             # on any final teardown
+            profiler = Profiler(
+                msg=f'Annotation teardown for ctx {ctx.cid}',
+                disabled=False,
+                ms_threshold=0.0,
+            )
             (_ctx, aids) = _ctxs[ctx.cid]
             assert _ctx is ctx
+            profiler(f'got {len(aids)} aids to remove')
+
             for aid in aids:
                 annot: QGraphicsItem = _annots[aid]
                 assert rm_annot(annot)
 
+            profiler(f'removed all {len(aids)} annotations')
+
 
 class AnnotCtl(Struct):
     '''
@@ -746,6 +1128,64 @@ class AnnotCtl(Struct):
                 )
             return aid
 
+    async def add_batch(
+        self,
+        fqme: str,
+        timeframe: float,
+        rects: list[dict]|None = None,
+        arrows: list[dict]|None = None,
+        texts: list[dict]|None = None,
+        show_individual_arrows: bool = False,
+
+        from_acm: bool = False,
+
+    ) -> dict[str, list[int]]:
+        '''
+        Batch submit multiple annotations in single IPC msg for
+        much faster remote annotation vs. per-annot round-trips.
+
+        Returns dict of annotation IDs:
+        {
+            'rects': [aid1, aid2, ...],
+            'arrows': [aid3, aid4, ...],
+            'texts': [aid5, aid6, ...],
+        }
+
+        '''
+        ipc: MsgStream = self._get_ipc(fqme)
+        with trio.fail_after(10):
+            await ipc.send({
+                'fqme': fqme,
+                'cmd': 'batch',
+                'timeframe': timeframe,
+                'rects': rects or [],
+                'arrows': arrows or [],
+                'texts': texts or [],
+                'show_individual_arrows': show_individual_arrows,
+            })
+            result: dict = await ipc.receive()
+            match result:
+                case {'error': str(msg)}:
+                    log.error(msg)
+                    return {
+                        'rects': [],
+                        'arrows': [],
+                        'texts': [],
+                    }
+
+            # register all AIDs with their IPC streams
+            for aid_list in result.values():
+                for aid in aid_list:
+                    self._ipcs[aid] = ipc
+                    if not from_acm:
+                        self._annot_stack.push_async_callback(
+                            partial(
+                                self.remove,
+                                aid,
+                            )
+                        )
+            return result
+
     async def add_text(
         self,
         fqme: str,
@@ -881,3 +1321,14 @@ async def open_annot_ctl(
                         _annot_stack=annots_stack,
                     )
                     yield client
+
+                    # client exited, measure teardown time
+                    teardown_profiler = Profiler(
+                        msg='Client AnnotCtl teardown',
+                        disabled=False,
+                        ms_threshold=0.0,
+                    )
+                    teardown_profiler('exiting annots_stack')
+
+                teardown_profiler('annots_stack exited')
+                teardown_profiler('exiting gather_contexts')

piker/ui/_search.py

@@ -174,6 +174,13 @@ class CompleterView(QTreeView):
 
         self.setStyleSheet(f"font: {size}px")
 
+    def update_fonts(self) -> None:
+        '''Update font sizes after zoom change.'''
+        self.set_font_size(_font.px_size)
+        self.setIndentation(_font.px_size)
+        self.setFont(_font.font)
+        self.updateGeometry()
+
     def resize_to_results(
         self,
         w: float | None = 0,
@@ -630,6 +637,29 @@ class SearchWidget(QtWidgets.QWidget):
             | align_flag.AlignLeft,
         )
 
+    def update_fonts(self) -> None:
+        '''Update font sizes after zoom change.'''
+        # regenerate label stylesheet with new font size
+        self.label.setStyleSheet(
+            f"""QLabel {{
+                color : {hcolor('default_lightest')};
+                font-size : {_font.px_size - 2}px;
+            }}
+            """
+        )
+        self.label.setFont(_font.font)
+
+        # update search bar and view fonts
+        if hasattr(self.bar, 'update_fonts'):
+            self.bar.update_fonts(_font)
+        elif hasattr(self.bar, 'setFont'):
+            self.bar.setFont(_font.font)
+
+        if hasattr(self.view, 'update_fonts'):
+            self.view.update_fonts()
+
+        self.updateGeometry()
+
     def focus(self) -> None:
         self.show()
         self.bar.focus()

piker/ui/_style.py

@@ -79,9 +79,13 @@ class DpiAwareFont:
         self._font_inches: float = None
         self._screen = None
 
-    def _set_qfont_px_size(self, px_size: int) -> None:
-        self._qfont.setPixelSize(px_size)
+    def _set_qfont_px_size(
+        self,
+        px_size: int,
+    ) -> int:
+        self._qfont.setPixelSize(int(px_size))
         self._qfm = QtGui.QFontMetrics(self._qfont)
+        return self.px_size
 
     @property
     def screen(self) -> QtGui.QScreen:
@@ -124,17 +128,22 @@ class DpiAwareFont:
 
         return size
 
-    def configure_to_dpi(self, screen: QtGui.QScreen | None = None):
+    def configure_to_dpi(
+        self,
+        screen: QtGui.QScreen | None = None,
+        zoom_level: float = 1.0,
+    ) -> int:
         '''
-        Set an appropriately sized font size depending on the screen DPI.
+        Set an appropriately sized font size depending on the screen DPI
+        or scale the size according to `zoom_level`.
 
-        If we end up needing to generalize this more here there are resources
-        listed in the script in ``snippets/qt_screen_info.py``.
+        If we end up needing to generalize this more here there are
+        resources listed in the script in
+        ``snippets/qt_screen_info.py``.
 
         '''
         if self._font_size is not None:
-            self._set_qfont_px_size(self._font_size)
-            return
+            return self._set_qfont_px_size(self._font_size * zoom_level)
 
         # NOTE: if no font size set either in the [ui] section of the
         # config or not yet computed from our magic scaling calcs,
@@ -153,7 +162,7 @@ class DpiAwareFont:
             ldpi = pdpi
 
         mx_dpi = max(pdpi, ldpi)
-        mn_dpi = min(pdpi, ldpi)
+        # mn_dpi = min(pdpi, ldpi)
         scale = round(ldpi/pdpi, ndigits=2)
 
         if mx_dpi <= 97:  # for low dpi use larger font sizes
@@ -162,7 +171,7 @@ class DpiAwareFont:
         else:  # hidpi use smaller font sizes
             inches = _font_sizes['hi'][self._font_size_calc_key]
 
-        dpi = mn_dpi
+        # dpi = mn_dpi
 
         mult = 1.0
 
@@ -197,24 +206,25 @@ class DpiAwareFont:
         # always going to hit that error in range mapping from inches:
         # float to px size: int.
         self._font_inches = inches
-        font_size = math.floor(inches * dpi)
+        font_size = math.floor(inches * pdpi)
+
+        # apply zoom level multiplier
+        font_size = int(font_size * zoom_level)
 
         log.debug(
             f"screen:{screen.name()}\n"
             f"pDPI: {pdpi}, lDPI: {ldpi}, scale: {scale}\n"
+            f"zoom_level: {zoom_level}\n"
             f"\nOur best guess font size is {font_size}\n"
         )
         # apply the size
-        self._set_qfont_px_size(font_size)
+        return self._set_qfont_px_size(font_size)
 
     def boundingRect(self, value: str) -> QtCore.QRectF:
-
-        screen = self.screen
-        if screen is None:
+        if self.screen is None:
             raise RuntimeError("You must call .configure_to_dpi() first!")
 
-        unscaled_br = self._qfm.boundingRect(value)
-
+        unscaled_br: QtCore.QRectF = self._qfm.boundingRect(value)
         return QtCore.QRectF(
             0,
             0,
@@ -228,12 +238,22 @@ _font = DpiAwareFont()
 _font_small = DpiAwareFont(_font_size_key='small')
 
 
-def _config_fonts_to_screen() -> None:
-    'configure global DPI aware font sizes'
+def _config_fonts_to_screen(
+    zoom_level: float = 1.0
+) -> int:
+    '''
+    Configure global DPI aware font size(s).
 
+    If `zoom_level` is provided we apply it to auto-calculated
+    DPI-aware font.
+
+    Return the new `DpiAwareFont.px_size`.
+
+    '''
     global _font, _font_small
-    _font.configure_to_dpi()
-    _font_small.configure_to_dpi()
+    _font.configure_to_dpi(zoom_level=zoom_level)
+    _font_small.configure_to_dpi(zoom_level=zoom_level)
+    return _font.px_size
 
 
 def get_fonts() -> tuple[