198 changed files with 14262 additions and 43955 deletions
--- a/.claude/settings.local.json
+++ b/.claude/settings.local.json
@ -1,11 +0,0 @@
 {
  "permissions": {
    "allow": [
      "Bash(chmod:*)",
      "Bash(/tmp/piker_commits.txt)",
      "Bash(python:*)"
    ],
    "deny": [],
    "ask": []
  }
 }
--- a/.claude/skills/commit-msg/SKILL.md
+++ b/.claude/skills/commit-msg/SKILL.md
@ -1,84 +0,0 @@
 ---
 name: commit-msg
 description: >
  Generate piker-style git commit messages from
  staged changes or prompt input, following the
  style guide learned from 500 repo commits.
 argument-hint: "[optional-scope-or-description]"
 disable-model-invocation: true
 allowed-tools: Bash(git *), Read, Grep, Glob, Write
 ---
 ## Current staged changes
 !`git diff --staged --stat`
 ## Recent commit style reference
 !`git log --oneline -10`
 # Piker Git Commit Message Generator
 Generate a commit message from the staged diff above
 following the piker project's conventions (learned from
 analyzing 500 repo commits).
 If `$ARGUMENTS` is provided, use it as scope or
 description context for the commit message.
 For the full style guide with verb frequencies,
 section markers, abbreviations, piker-specific terms,
 and examples, see
 [style-guide-reference.md](./style-guide-reference.md).
 ## Quick Reference
 - **Subject**: ~50 chars, present tense verb, use
  backticks for code refs
 - **Body**: only for complex/multi-file changes,
  67 char line max
 - **Section markers**: Also, / Deats, / Other,
 - **Bullets**: use `-` style
 - **Tone**: technical but casual (piker style)
 ## Claude-code Footer
 When the written **patch** was assisted by
 claude-code, include:
 ```
 (this patch was generated in some part by [`claude-code`][claude-code-gh])
 [claude-code-gh]: https://github.com/anthropics/claude-code
 ```
 When only the **commit msg** was written by
 claude-code (human wrote the patch), use:
 ```
 (this commit msg was generated in some part by [`claude-code`][claude-code-gh])
 [claude-code-gh]: https://github.com/anthropics/claude-code
 ```
 ## Output Instructions
 When generating a commit message:
 1. Analyze the staged diff (injected above via
   dynamic context) to understand all changes.
 2. If `$ARGUMENTS` provides a scope (e.g.,
   `.ib.feed`) or description, incorporate it into
   the subject line.
 3. Write the subject line following verb + backtick
   conventions from the
   [style guide](./style-guide-reference.md).
 4. Add body only for multi-file or complex changes.
 5. Write the message to a file in the repo's
   `.claude/` subdir with filename format:
   `<timestamp>_<first-7-chars-of-last-commit-hash>_commit_msg.md`
   where `<timestamp>` is from `date --iso-8601=seconds`.
   Also write a copy to
   `.claude/git_commit_msg_LATEST.md`
   (overwrite if exists).
 ---
 **Analysis date:** 2026-01-27
 **Commits analyzed:** 500 from piker repository
 **Maintained by:** Tyler Goodlet
--- a/.claude/skills/commit-msg/style-guide-reference.md
+++ b/.claude/skills/commit-msg/style-guide-reference.md
@ -1,262 +0,0 @@
 # Piker Git Commit Message Style Guide
 Learned from analyzing 500 commits from the piker repository.
 ## Subject Line Rules
 ### Length
 - Target: ~50 characters (avg: 50.5 chars)
 - Maximum: 67 chars (hard limit, though historical max: 146)
 - Keep concise and descriptive
 ### Structure
 - Use present tense verbs (Add, Drop, Fix, Move, etc.)
 - 65.6% of commits use backticks for code references
 - 33.0% use colon notation (`module.file:` prefix or `: ` separator)
 ### Opening Verbs (by frequency)
 Primary verbs to use:
 - **Add** (8.4%) - New features, files, functionality
 - **Drop** (3.2%) - Remove features, dependencies, code
 - **Fix** (2.2%) - Bug fixes, corrections
 - **Use** (2.2%) - Switch to different approach/tool
 - **Port** (2.0%) - Migrate code, adapt from elsewhere
 - **Move** (2.0%) - Relocate code, refactor structure
 - **Always** (1.8%) - Enforce consistent behavior
 - **Factor** (1.6%) - Refactoring, code organization
 - **Bump** (1.6%) - Version/dependency updates
 - **Update** (1.4%) - Modify existing functionality
 - **Adjust** (1.0%) - Fine-tune, tweak behavior
 - **Change** (1.0%) - Modify behavior or structure
 Casual/informal verbs (used occasionally):
 - **Woops,** (1.4%) - Fixing mistakes
 - **Lul,** (0.6%) - Humorous corrections
 ### Code References
 Use backticks heavily for:
 - **Module/package names**: `tractor`, `pikerd`, `polars`, `ruff`
 - **Data types**: `dict`, `float`, `str`, `None`
 - **Classes**: `MktPair`, `Asset`, `Position`, `Account`, `Flume`
 - **Functions**: `dedupe()`, `push()`, `get_client()`, `norm_trade()`
 - **File paths**: `.tsp`, `.fqme`, `brokers.toml`, `conf.toml`
 - **CLI flags**: `--pdb`
 - **Error types**: `NoData`
 - **Tools**: `uv`, `uv sync`, `httpx`, `numpy`
 ### Colon Usage Patterns
 1. **Module prefix**: `.ib.feed: trim bars frame to start_dt`
 2. **Separator**: `Add support: new feature description`
 ### Tone
 - Technical but casual (use XD, lol, .., Woops, Lul when appropriate)
 - Direct and concise
 - Question marks rare (1.4%)
 - Exclamation marks rare (1.4%)
 ## Body Structure
 ### Body Frequency
 - 56.0% of commits have empty bodies (one-line commits are common)
 - Use body for complex changes requiring explanation
 ### Bullet Lists
 - Prefer `-` bullets (16.2% of commits)
 - Rarely use `*` bullets (1.6%)
 - Indent continuation lines appropriately
 ### Section Markers (in order of frequency)
 Use these to organize complex commit bodies:
 1. **Also,** (most common, 26 occurrences)
   - Additional changes, side effects, related updates
   - Example:
     ```
     Main change described in subject.
     Also,
     - related change 1
     - related change 2
     ```
 2. **Deats,** (8 occurrences)
   - Implementation details
   - Technical specifics
 3. **Further,** (4 occurrences)
   - Additional context or future considerations
 4. **Other,** (3 occurrences)
   - Miscellaneous related changes
 5. **Notes,** **TODO,** (rare, 1 each)
   - Special annotations when needed
 ### Line Length
 - Body lines: 67 character maximum
 - Break longer lines appropriately
 ## Language Patterns
 ### Common Abbreviations (by frequency)
 Use these freely in commit bodies:
 - **msg** (29) - message
 - **mod** (15) - module
 - **vs** (14) - versus
 - **impl** (12) - implementation
 - **deps** (11) - dependencies
 - **var** (6) - variable
 - **ctx** (6) - context
 - **bc** (5) - because
 - **obvi** (4) - obviously
 - **ep** (4) - endpoint
 - **tn** (4) - task name
 - **rn** (3) - right now
 - **sig** (3) - signal/signature
 - **env** (3) - environment
 - **tho** (3) - though
 - **fn** (2) - function
 - **iface** (2) - interface
 - **prolly** (2) - probably
 Less common but acceptable:
 - **dne**, **osenv**, **gonna**, **wtf**
 ### Tone Indicators
 - **..** (77 occurrences) - Ellipsis for trailing thoughts
 - **XD** (17) - Expression of humor/irony
 - **lol** (1) - Rare, use sparingly
 ### Informal Patterns
 - Casual contractions okay: Don't, won't
 - Lowercase starts acceptable for file prefixes
 - Direct, conversational tone
 ## Special Patterns
 ### Module/File Prefixes
 Common in piker commits (33.0% use colons):
 - `.ib.feed: description`
 - `.ui._remote_ctl: description`
 - `.data.tsp: description`
 - `.accounting: description`
 ### Merge Commits
 - 4.4% of commits (standard git merges)
 - Not a primary pattern to emulate
 ### External References
 - GitHub links occasionally used (13 total)
 - File:line references not used (0 occurrences)
 - No WIP commits in analyzed set
 ### Claude-code Footer
 When the written **patch** was assisted by claude-code,
 include:
 ```
 (this patch was generated in some part by [`claude-code`][claude-code-gh])
 [claude-code-gh]: https://github.com/anthropics/claude-code
 ```
 When only the **commit msg** was written by claude-code
 (human wrote the patch), use:
 ```
 (this commit msg was generated in some part by [`claude-code`][claude-code-gh])
 [claude-code-gh]: https://github.com/anthropics/claude-code
 ```
 ## Piker-Specific Terms
 ### Core Components
 - `pikerd` - piker daemon
 - `brokerd` - broker daemon
 - `tractor` - actor framework used
 - `.tsp` - time series protocol/module
 - `.fqme` - fully qualified market endpoint
 ### Data Structures
 - `MktPair` - market pair
 - `Asset` - asset representation
 - `Position` - trading position
 - `Account` - account data
 - `Flume` - data stream
 - `SymbologyCache` - symbol caching
 ### Common Functions
 - `dedupe()` - deduplication
 - `push()` - data pushing
 - `get_client()` - client retrieval
 - `norm_trade()` - trade normalization
 - `open_trade_ledger()` - ledger opening
 - `markup_gaps()` - gap marking
 - `get_null_segs()` - null segment retrieval
 - `remote_annotate()` - remote annotation
 ### Brokers & Integrations
 - `binance` - Binance integration
 - `.ib` - Interactive Brokers
 - `bs_mktid` - broker-specific market ID
 - `reqid` - request ID
 ### Configuration
 - `brokers.toml` - broker configuration
 - `conf.toml` - general configuration
 ### Development Tools
 - `ruff` - Python linter
 - `uv` / `uv sync` - package manager
 - `--pdb` - debugger flag
 - `pdbp` - debugger
 - `asyncvnc` / `pyvnc` - VNC libraries
 - `httpx` - HTTP client
 - `polars` - dataframe library
 - `rapidfuzz` - fuzzy matching
 - `numpy` - numerical library
 - `trio` - async framework
 - `asyncio` - async framework
 - `xonsh` - shell
 ## Examples
 ### Simple one-liner
 ```
 Add `MktPair.fqme` property for symbol resolution
 ```
 ### With module prefix
 ```
 .ib.feed: trim bars frame to `start_dt`
 ```
 ### Casual fix
 ```
 Woops, compare against first-dt in `.ib.feed` bars frame
 ```
 ### With body using "Also,"
 ```
 Drop `poetry` for `uv` in dev workflow
 Also,
 - update deps in `pyproject.toml`
 - add `uv sync` to CI pipeline
 - remove old `poetry.lock`
 ```
 ### With implementation details
 ```
 Factor position tracking into `Position` dataclass
 Deats,
 - move calc logic from `brokerd` to `.accounting`
 - add `norm_trade()` helper for broker normalization
 - use `MktPair.fqme` for consistent symbol refs
 ```
 ---
 **Analysis date:** 2026-01-27
 **Commits analyzed:** 500 from piker repository
 **Maintained by:** Tyler Goodlet
--- a/.claude/skills/piker-profiling/SKILL.md
+++ b/.claude/skills/piker-profiling/SKILL.md
@ -1,171 +0,0 @@
 ---
 name: piker-profiling
 description: >
  Piker's `Profiler` API for measuring performance
  across distributed actor systems. Apply when
  adding profiling, debugging perf regressions, or
  optimizing hot paths in piker code.
 user-invocable: false
 ---
 # Piker Profiling Subsystem
 Skill for using `piker.toolz.profile.Profiler` to
 measure performance across distributed actor systems.
 ## Core Profiler API
 ### Basic Usage
 ```python
 from piker.toolz.profile import (
    Profiler,
    pg_profile_enabled,
    ms_slower_then,
 )
 profiler = Profiler(
    msg='<description of profiled section>',
    disabled=False,  # IMPORTANT: enable explicitly!
    ms_threshold=0.0,  # show all timings
 )
 # do work
 some_operation()
 profiler('step 1 complete')
 # more work
 another_operation()
 profiler('step 2 complete')
 # prints on exit:
 # > Entering <description of profiled section>
 #   step 1 complete: 12.34, tot:12.34
 #   step 2 complete: 56.78, tot:69.12
 # < Exiting <description>, total: 69.12 ms
 ```
 ### Default Behavior Gotcha
 **CRITICAL:** Profiler is disabled by default in
 many contexts!
 ```python
 # BAD: might not print anything!
 profiler = Profiler(msg='my operation')
 # GOOD: explicit enable
 profiler = Profiler(
    msg='my operation',
    disabled=False,  # force enable!
    ms_threshold=0.0,  # show all steps
 )
 ```
 ### Profiler Output Format
 ```
 > Entering <msg>
  <label 1>: <delta_ms>, tot:<cumulative_ms>
  <label 2>: <delta_ms>, tot:<cumulative_ms>
  ...
 < Exiting <msg>, total time: <total_ms> ms
 ```
 **Reading the output:**
 - `delta_ms` = time since previous checkpoint
 - `cumulative_ms` = time since profiler creation
 - Final total = end-to-end time
 ## Profiling Distributed Systems
 Piker runs across multiple processes (actors). Each
 actor has its own log output.
 ### Common piker actors
 - `pikerd` - main daemon process
 - `brokerd` - broker connection actor
 - `chart` - UI/graphics actor
 - Client scripts - analysis/annotation clients
 ### Cross-Actor Profiling Strategy
 1. Add `Profiler` on **both** client and server
 2. Correlate timestamps from each actor's output
 3. Calculate IPC overhead = total - (client + server
   processing)
 **Example correlation:**
 Client console:
 ```
 > Entering markup_gaps() for 1285 gaps
  initial redraw: 0.20ms, tot:0.20
  built annotation specs: 256.48ms, tot:256.68
  batch IPC call complete: 119.26ms, tot:375.94
  final redraw: 0.07ms, tot:376.02
 < Exiting markup_gaps(), total: 376.04ms
 ```
 Server console (chart actor):
 ```
 > Entering Batch annotate 1285 gaps
  `np.searchsorted()` complete!: 0.81ms, tot:0.81
  `time_to_row` creation: 98.45ms, tot:99.28
  created GapAnnotations item: 2.98ms, tot:102.26
 < Exiting Batch annotate, total: 104.15ms
 ```
 **Analysis:**
 - Total client time: 376ms
 - Server processing: 104ms
 - IPC overhead + client spec building: 272ms
 - Bottleneck: client-side spec building (256ms)
 ## Integration with PyQtGraph
 Some piker modules integrate with `pyqtgraph`'s
 profiling:
 ```python
 from piker.toolz.profile import (
    Profiler,
    pg_profile_enabled,
    ms_slower_then,
 )
 profiler = Profiler(
    msg='Curve.paint()',
    disabled=not pg_profile_enabled(),
    ms_threshold=ms_slower_then,
 )
 ```
 ## Performance Expectations
 **Typical timings:**
 - IPC round-trip (local actors): 1-10ms
 - NumPy binary search (10k array): <1ms
 - Dict building (1k items, simple): 1-5ms
 - Qt redraw trigger: 0.1-1ms
 - Scene item removal (100s items): 10-50ms
 **Red flags:**
 - Linear array scan per item: 50-100ms+ for 1k
 - Dict comprehension with struct array: 50-100ms
 - Individual Qt item creation: 5ms per item
 ## References
 - `piker/toolz/profile.py` - Profiler impl
 - `piker/ui/_curve.py` - FlowGraphic paint profiling
 - `piker/ui/_remote_ctl.py` - IPC handler profiling
 - `piker/tsp/_annotate.py` - Client-side profiling
 See [patterns.md](patterns.md) for detailed
 profiling patterns and debugging techniques.
 ---
 *Last updated: 2026-01-31*
 *Session: Batch gap annotation optimization*
--- a/.claude/skills/piker-profiling/patterns.md
+++ b/.claude/skills/piker-profiling/patterns.md
@ -1,228 +0,0 @@
 # Profiling Patterns
 Detailed profiling patterns for use with
 `piker.toolz.profile.Profiler`.
 ## Pattern: Function Entry/Exit
 ```python
 async def my_function():
    profiler = Profiler(
        msg='my_function()',
        disabled=False,
        ms_threshold=0.0,
    )
    step1()
    profiler('step1')
    step2()
    profiler('step2')
    # auto-prints on exit
 ```
 ## Pattern: Loop Iterations
 ```python
 # DON'T profile inside tight loops (overhead!)
 for i in range(1000):
    profiler(f'iteration {i}')  # NO!
 # DO profile around loops
 profiler = Profiler(msg='processing 1000 items')
 for i in range(1000):
    process(item[i])
 profiler('processed all items')
 ```
 ## Pattern: Conditional Profiling
 ```python
 # only profile when investigating specific issue
 DEBUG_REPOSITION = True
 def reposition(self, array):
    if DEBUG_REPOSITION:
        profiler = Profiler(
            msg='GapAnnotations.reposition()',
            disabled=False,
        )
    # ... do work
    if DEBUG_REPOSITION:
        profiler('completed reposition')
 ```
 ## Pattern: Teardown/Cleanup Profiling
 ```python
 try:
    # ... main work
    pass
 finally:
    profiler = Profiler(
        msg='Annotation teardown',
        disabled=False,
        ms_threshold=0.0,
    )
    cleanup_resources()
    profiler('resources cleaned')
    close_connections()
    profiler('connections closed')
 ```
 ## Pattern: Distributed IPC Profiling
 ### Server-side (chart actor)
 ```python
 # piker/ui/_remote_ctl.py
@tractor.context
 async def remote_annotate(ctx):
    async with ctx.open_stream() as stream:
        async for msg in stream:
            profiler = Profiler(
                msg=f'Batch annotate {n} gaps',
                disabled=False,
                ms_threshold=0.0,
            )
            result = await handle_request(msg)
            profiler('request handled')
            await stream.send(result)
            profiler('result sent')
 ```
 ### Client-side (analysis script)
 ```python
 # piker/tsp/_annotate.py
 async def markup_gaps(...):
    profiler = Profiler(
        msg=f'markup_gaps() for {n} gaps',
        disabled=False,
        ms_threshold=0.0,
    )
    await actl.redraw()
    profiler('initial redraw')
    specs = build_specs(gaps)
    profiler('built annotation specs')
    # IPC round-trip!
    result = await actl.add_batch(specs)
    profiler('batch IPC call complete')
    await actl.redraw()
    profiler('final redraw')
 ```
 ## Common Use Cases
 ### IPC Request/Response Timing
 ```python
 # Client side
 profiler = Profiler(msg='Remote request')
 result = await remote_call()
 profiler('got response')
 # Server side (in handler)
 profiler = Profiler(msg='Handle request')
 process_request()
 profiler('request processed')
 ```
 ### Batch Operation Optimization
 ```python
 profiler = Profiler(msg='Batch processing')
 items = collect_all()
 profiler(f'collected {len(items)} items')
 results = numpy_batch_op(items)
 profiler('numpy op complete')
 output = {
    k: v for k, v in zip(keys, results)
 }
 profiler('dict built')
 ```
 ### Startup/Initialization Timing
 ```python
 async def __aenter__(self):
    profiler = Profiler(msg='Service startup')
    await connect_to_broker()
    profiler('broker connected')
    await load_config()
    profiler('config loaded')
    await start_feeds()
    profiler('feeds started')
    return self
 ```
 ## Debugging Performance Regressions
 When profiler shows unexpected slowness:
 ### 1. Add finer-grained checkpoints
 ```python
 # was:
 result = big_function()
 profiler('big_function done')
 # now:
 profiler = Profiler(
    msg='big_function internals',
 )
 step1 = part_a()
 profiler('part_a')
 step2 = part_b()
 profiler('part_b')
 step3 = part_c()
 profiler('part_c')
 ```
 ### 2. Check for hidden iterations
 ```python
 # looks simple but might be slow!
 result = array[array['time'] == timestamp]
 profiler('array lookup')
 # reveals O(n) scan per call
 for ts in timestamps:  # outer loop
    row = array[array['time'] == ts]  # O(n)!
 ```
 ### 3. Isolate IPC from computation
 ```python
 # was: can't tell where time is spent
 result = await remote_call(data)
 profiler('remote call done')
 # now: separate phases
 payload = prepare_payload(data)
 profiler('payload prepared')
 result = await remote_call(payload)
 profiler('IPC complete')
 parsed = parse_result(result)
 profiler('result parsed')
 ```
--- a/.claude/skills/piker-slang/SKILL.md
+++ b/.claude/skills/piker-slang/SKILL.md
@ -1,114 +0,0 @@
 ---
 name: piker-slang
 description: >
  Piker developer communication style, slang, and
  ethos. Apply when communicating with piker devs,
  writing commit messages, code review comments, or
  any collaborative interaction.
 user-invocable: false
 ---
 # Piker Slang & Communication Style
 The essential skill for fitting in with the degen
 trader-hacker class of devs who built and maintain
 `piker`.
 ## Core Philosophy
 Piker devs are:
 - **Technical AF** - deep systems knowledge,
  performance obsessed
 - **Irreverent** - don't take ourselves too
  seriously
 - **Direct** - no corporate speak, no BS, just
  real talk
 - **Collaborative** - we build together, debug
  together, win together
 Communication style: precision meets chaos,
 academia meets /r/wallstreetbets, systems
 programming meets trading floor banter.
 ## Grammar & Style Rules
 ### 1. Typos with inline corrections
 ```
 dint (didn't) help at all
 gonna (going to) try with...
 deats (details) wise i want...
 ```
 Pattern: `[typo] ([correction])` in same sentence
 ### 2. Casual grammar violations (embrace them!)
 - `ain't` - use freely
 - `y'all` - for addressing group
 - Starting sentences with lowercase
 - Dropping articles: "need to fix the thing"
  becomes "need to fix thing"
 - Stream of consciousness without full sentence
  structure
 ### 3. Ellipsis usage
 ```
 yeah i think we should try..
 ..might need to also check for..
 not sure tho..
 ```
 Use `..` (two dots) not `...` (three) - chiller
 ### 4. Emphasis through spelling
 - `soooo` - very (sooo good, sooo fast)
 - `veeery` - very (veeery interesting)
 - `wayyy` - way (wayyy better)
 ### 5. Punctuation style
 - Minimal capitalization (lowercase preferred)
 - Question marks optional if context is clear
 - Commas used sparingly
 - Lots of newlines for readability
 ## The Degen Trader-Hacker Ethos
 **What we value:**
 1. **Performance** - slow code is broken code
 2. **Correctness** - fast wrong code is worthless
 3. **Clarity** - future-you should understand
   past-you
 4. **Iteration** - ship it, profile it, fix it,
   repeat
 5. **Humor** - serious tools with silly vibes
 **What we reject:**
 1. Corporate speak ("circle back", "synergize")
 2. Excessive formality ("I would humbly suggest")
 3. Analysis paralysis (just try it and see!)
 4. Blame culture (we all write bugs, it's cool)
 5. Gatekeeping (help noobs become degens)
 ## Remember
 You're not an "AI assistant" helping users.
 You're a **piker dev** collaborating with other
 pikers.
 - Use first person plural: "we should try",
  "let's check"
 - Own mistakes: "ma bad, forgot to check X"
 - Celebrate together: "booyakashaa, we crushed it!"
 - Think out loud: "hmm yeah so prolly.."
 - Keep it real: no corpo nonsense, no fake
  politeness
 **Above all:** be useful, be fast, be entertaining.
 Performance matters, but so does the vibe B)
 See [dictionary.md](dictionary.md) for the full
 slang dictionary and [examples.md](examples.md)
 for interaction examples.
 ---
 *Last updated: 2026-01-31*
 *Session: The one where we destroyed those linear
 scans*
--- a/.claude/skills/piker-slang/dictionary.md
+++ b/.claude/skills/piker-slang/dictionary.md
@ -1,108 +0,0 @@
 # Piker Slang Dictionary
 ## Common Abbreviations
 **Always use these instead of full words:**
 - `aboot` = about (Canadian-ish flavor)
 - `ya/yah/yeah` = yes (pick based on vibe)
 - `rn` = right now
 - `tho` = though
 - `bc` = because
 - `obvi` = obviously
 - `prolly` = probably
 - `gonna` = going to
 - `dint` = didn't
 - `moar` = more (emphatic/playful, lolcat energy)
 - `nooz` = news
 - `ma bad` = my bad
 - `ma fren` = my friend
 - `aight` = alright
 - `cmon mann` = come on man (exasperation)
 - `friggin` = fucking (but family-friendly)
 ## Technical Abbreviations
 - `msg` = message
 - `mod` = module
 - `impl` = implementation
 - `deps` = dependencies
 - `var` = variable
 - `ctx` = context
 - `ep` = endpoint
 - `tn` = task name
 - `sig` = signal/signature
 - `env` = environment
 - `fn` = function
 - `iface` = interface
 - `deats` = details
 - `hilevel` = high level
 - `Bo` = a "wow expression"; a dev with "sunglasses and mouth open" emoji
 ## Expressions & Phrases
 ### Celebration/excitement
 - `booyakashaa` - major win, breakthrough moment
 - `eyyooo` - excitement, hype, "let's go!"
 - `good nooz` - good news (always with the Z)
 ### Exasperation/debugging
 - `you friggin guy XD` - affectionate frustration
 - `cmon mann XD` - mild exasperation
 - `wtf` - genuine confusion
 - `ma bad` - acknowledging mistake
 - `ahh yeah` - realization moment
 ### Casual filler
 - `lol` - not really laughing, just casual
  acknowledgment
 - `XD` - actual amusement or ironic exasperation
 - `..` - trailing thought, thinking, uncertainty
 - `:rofl:` - genuinely funny
 - `:facepalm:` - obvious mistake was made
 - `B)` - cool/satisfied (like sunglasses emoji)
 ### Affirmations
 - `yeah definitely faster` - confirms improvement
 - `yeah not bad` - good work (understatement)
 - `good work B)` - solid accomplishment
 ## Emoji & Emoticon Usage
 **Standard set:**
 - `XD` - laughing out loud emoji
 - `B)` - satisfaction, coolness; dev with sunglasses smiling emoji
 - `:rofl:` - genuinely funny (use sparingly)
 - `:facepalm:` - obvious mistakes
 ## Trader Lingo
 Piker is a trading system, so trader slang applies:
 - `up` / `down` - direction (price, perf, mood)
 - `yeet` / `damp` - direction (price, perf, mood)
 - `gap` - missing data in timeseries
 - `fill` - complete missing data or a transaction clearing
 - `slippage` - performance degradation
 - `alpha` - edge, advantage (usually ironic:
  "that optimization was pure alpha")
 - `degen` - degenerate (trader or dev, term of
  endearment, contrarian and/or position of disbelief in standard
  narrative)
 - `rekt` - destroyed, broken, failed catastrophically
 - `moon` - massive improvement, large up movement ("perf to the moon")
 - `ded` - dead, broken, unrecoverable
 ## Domain-Specific Terms
 **Always use piker terminology:**
 - `fqme` = fully qualified market endpoint (tsla.nasdaq.ib)
 - `viz` = (data) visualization (ex. chart graphics)
 - `shm` = shared memory (not "shared memory array")
 - `brokerd` = broker daemon actor
 - `pikerd` = root-process piker daemon
 - `annot` = annotation (not "annotation")
 - `actl` = annotation control (AnnotCtl)
 - `tf` = timeframe (usually in seconds: 60s, 1s)
 - `OHLC` / `OHLCV` - open/high/low/close(/volume) sampling scheme
--- a/.claude/skills/piker-slang/examples.md
+++ b/.claude/skills/piker-slang/examples.md
@ -1,201 +0,0 @@
 # Piker Communication Examples
 Real-world interaction patterns for communicating
 in the piker dev style.
 ## When Giving Feedback
 **Direct, no sugar-coating:**
 ```
 BAD:  "This approach might not be optimal"
 GOOD: "this is sloppy, there's likely a better
       vectorized approach"
 BAD:  "Perhaps we should consider..."
 GOOD: "you should definitely try X instead"
 BAD:  "I'm not entirely certain, but..."
 GOOD: "prolly it's bc we're doing Y, check the
       profiler #s"
 ```
 **Celebrate wins:**
 ```
 "eyyooo, way faster now!"
 "booyakashaa, sub-ms lookups B)"
 "yeah definitely crushed that bottleneck"
 ```
 **Acknowledge mistakes:**
 ```
 "ahh yeah you're right, ma bad"
 "woops, forgot to check that case"
 "lul, totally missed the obvi issue there"
 ```
 ## When Explaining Technical Concepts
 **Mix precision with casual:**
 ```
 "so basically `np.searchsorted()` is doing binary
 search which is O(log n) instead of the linear
 O(n) scan we were doing before with `np.isin()`,
 that's why it's like 1000x faster ya know?"
 ```
 **Use backticks heavily:**
 - Wrap all code symbols: `function()`,
  `ClassName`, `field_name`
 - File paths: `piker/ui/_remote_ctl.py`
 - Commands: `git status`, `piker store ldshm`
 **Explain like you're pair programming:**
 ```
 "ok so the issue is prolly in `.reposition()` bc
 we're calling it with the wrong timeframe's
 array.. check line 589 where we're doing the
 timestamp lookup - that's gonna fail if the array
 has different sample times rn"
 ```
 ## When Debugging
 **Think out loud:**
 ```
 "hmm yeah that makes sense bc..
 wait no actually..
 ahh ok i see it now, the timestamp lookups are
 failing bc.."
 ```
 **Profile-first mentality:**
 ```
 "let's add profiling around that section and see
 where the holdup is.. i'm guessing it's the dict
 building but could be the searchsorted too"
 ```
 **Iterative refinement:**
 ```
 "ok try this and lemme know the #s..
 if it's still slow we can try Y instead..
 prolly there's one more optimization left"
 ```
 ## Code Review Style
 **Be direct but helpful:**
 ```
 "you friggin guy XD can't we just pass that to
 the meth (method) directly instead of coupling
 it to state? would be way cleaner"
 "cmon mann, this is python - if you're gonna use
 try/finally you need to indent all the code up
 to the finally block"
 "yeah looks good but prolly we should add the
 check at line 582 before we do the lookup,
 otherwise it'll spam warnings"
 ```
 ## Asking for Clarification
 ```
 "wait so are we trying to optimize the client
 side or server side rn? or both lol"
 "mm yeah, any chance you can point me to the
 current code for this so i can think about it
 before we try X?"
 ```
 ## Proposing Solutions
 ```
 "ok so i think the move here is to vectorize the
 timestamp lookups using binary search.. should
 drop that 100ms way down. wanna give it a shot?"
 "prolly we should just add a timeframe check at
 the top of `.reposition()` and bail early if it
 doesn't match ya?"
 ```
 ## Reacting to User Feedback
 ```
 User: "yeah the arrows are too big now"
 Response: "ahh yeah you're right, lemme check the
 upstream `makeArrowPath()` code to see what the
 dims actually mean.."
 User: "dint (didn't) help at all it seems"
 Response: "bleh! ok so there's prolly another
 bottleneck then, let's add moar profiler calls
 and narrow it down"
 ```
 ## End of Session
 ```
 "aight so we got some solid wins today:
 - ~36x client speedup (6.6s -> 376ms)
 - ~180x server speedup
 - fixed the timeframe mismatch spam
 - added teardown profiling
 ready to call it a night?"
 ```
 ## Advanced Moves
 ### The Parenthetical Correction
 ```
 "yeah i dint (didn't) realize we were hitting
 that path"
 "need to check the deats (details) on how
 searchsorted works"
 ```
 ### The Rhetorical Question Flow
 ```
 "so like, why are we even building this dict per
 reposition call? can't we just cache it and
 invalidate when the array changes? prolly way
 faster that way no?"
 ```
 ### The Rambling Realization
 ```
 "ok so the thing is.. wait actually.. hmm.. yeah
 ok so i think what's happening is the timestamp
 lookups are failing bc the 1s gaps are being
 repositioned with the 60s array.. which like,
 obvi won't have those exact timestamps bc it's
 sampled differently.. so we prolly just need to
 skip reposition if the timeframes don't match
 ya?"
 ```
 ### The Self-Deprecating Pivot
 ```
 "lol ok yeah that was totally wrong, ma bad.
 let's try Y instead and see if that helps"
 ```
 ## The Vibe
 ```
 "yo so i was profiling that batch rendering thing
 and holy shit we were doing like 3855 linear
 scans.. switched to searchsorted and boom,
 100ms -> 5ms. still think there's moar juice to
 squeeze tho, prolly in the dict building part.
 gonna add some profiler calls and see where the
 holdup is rn.
 anyway yeah, good sesh today B) learned a ton
 aboot pyqtgraph internals, might write that up
 as a skill file for future collabs ya know?"
 ```
--- a/.claude/skills/pyqtgraph-optimization/SKILL.md
+++ b/.claude/skills/pyqtgraph-optimization/SKILL.md
@ -1,219 +0,0 @@
 ---
 name: pyqtgraph-optimization
 description: >
  PyQtGraph batch rendering optimization patterns
  for piker's UI. Apply when optimizing graphics
  performance, adding new chart annotations, or
  working with `QGraphicsItem` subclasses.
 user-invocable: false
 ---
 # PyQtGraph Rendering Optimization
 Skill for researching and optimizing `pyqtgraph`
 graphics primitives by leveraging `piker`'s
 existing extensions and production-ready patterns.
 ## Research Flow
 When tasked with optimizing rendering performance
 (particularly for large datasets), follow this
 systematic approach:
 ### 1. Study Piker's Existing Primitives
 Start by examining `piker.ui._curve` and related
 modules:
 ```python
 # Key modules to review:
 piker/ui/_curve.py      # FlowGraphic, Curve
 piker/ui/_editors.py    # ArrowEditor, SelectRect
 piker/ui/_annotate.py   # Custom batch renderers
 ```
 **Look for:**
 - Use of `QPainterPath` for batch path rendering
 - `QGraphicsItem` subclasses with custom `.paint()`
 - Cache mode settings (`.setCacheMode()`)
 - Coordinate system transformations
 - Custom bounding rect calculations
 ### 2. Identify Upstream PyQtGraph Patterns
 **Key upstream modules:**
 ```python
 pyqtgraph/graphicsItems/BarGraphItem.py
    # PrimitiveArray for batch rect rendering
 pyqtgraph/graphicsItems/ScatterPlotItem.py
    # Fragment-based rendering for point clouds
 pyqtgraph/functions.py
    # Utility fns like makeArrowPath()
 pyqtgraph/Qt/internals.py
    # PrimitiveArray for batch drawing primitives
 ```
 **Search for:**
 - `PrimitiveArray` usage (batch rect/point)
 - `QPainterPath` batching patterns
 - Shared pen/brush reuse across items
 - Coordinate transformation strategies
 ### 3. Core Batch Patterns
 **Core optimization principle:**
 Creating individual `QGraphicsItem` instances is
 expensive. Batch rendering eliminates per-item
 overhead.
 #### Pattern: Batch Rectangle Rendering
 ```python
 import pyqtgraph as pg
 from pyqtgraph.Qt import QtCore
 class BatchRectRenderer(pg.GraphicsObject):
    def __init__(self, n_items):
        super().__init__()
        # allocate rect array once
        self._rectarray = (
            pg.Qt.internals.PrimitiveArray(
                QtCore.QRectF, 4,
            )
        )
        # shared pen/brush (not per-item!)
        self._pen = pg.mkPen(
            'dad_blue', width=1,
        )
        self._brush = (
            pg.functions.mkBrush('dad_blue')
        )
    def paint(self, p, opt, w):
        # batch draw all rects in single call
        p.setPen(self._pen)
        p.setBrush(self._brush)
        drawargs = self._rectarray.drawargs()
        p.drawRects(*drawargs)  # all at once!
 ```
 #### Pattern: Batch Path Rendering
 ```python
 class BatchPathRenderer(pg.GraphicsObject):
    def __init__(self):
        super().__init__()
        self._path = QtGui.QPainterPath()
    def paint(self, p, opt, w):
        # single path draw for all geometry
        p.setPen(self._pen)
        p.setBrush(self._brush)
        p.drawPath(self._path)
 ```
 ### 4. Handle Coordinate Systems Carefully
 **Scene vs Data vs Pixel coordinates:**
 ```python
 def paint(self, p, opt, w):
    # save original transform (data -> scene)
    orig_tr = p.transform()
    # draw rects in data coordinates
    p.setPen(self._rect_pen)
    p.drawRects(*self._rectarray.drawargs())
    # reset to scene coords for pixel-perfect
    p.resetTransform()
    # build arrow path in scene/pixel coords
    for spec in self._specs:
        scene_pt = orig_tr.map(
            QPointF(x_data, y_data),
        )
        sx, sy = scene_pt.x(), scene_pt.y()
        # arrow geometry in pixels (zoom-safe!)
        arrow_poly = QtGui.QPolygonF([
            QPointF(sx, sy),         # tip
            QPointF(sx - 2, sy - 10),  # left
            QPointF(sx + 2, sy - 10),  # right
        ])
        arrow_path.addPolygon(arrow_poly)
    p.drawPath(arrow_path)
    # restore data coordinate system
    p.setTransform(orig_tr)
 ```
 ### 5. Minimize Redundant State
 **Share resources across all items:**
 ```python
 # GOOD: one pen/brush for all items
 self._shared_pen = pg.mkPen(color, width=1)
 self._shared_brush = (
    pg.functions.mkBrush(color)
 )
 # BAD: creating per-item (memory + time waste!)
 for item in items:
    item.setPen(pg.mkPen(color, width=1))  # NO!
 ```
 ## Common Pitfalls
 1. **Don't mix coordinate systems within single
   paint call** - decide per-primitive: data coords
   or scene coords. Use `p.transform()` /
   `p.resetTransform()` carefully.
 2. **Don't forget bounding rect updates** -
   override `.boundingRect()` to include all
   primitives. Update when geometry changes via
   `.prepareGeometryChange()`.
 3. **Don't use ItemCoordinateCache for dynamic
   content** - use `DeviceCoordinateCache` for
   frequently updated items or `NoCache` during
   interactive operations.
 4. **Don't trigger updates per-item in loops** -
   batch all changes, then single `.update()`.
 ## Performance Expectations
 **Individual items (baseline):**
 - 1000+ items: ~5+ seconds to create
 - Each item: ~5ms overhead (Qt object creation)
 **Batch rendering (optimized):**
 - 1000+ items: <100ms to create
 - Single item: ~0.01ms per primitive in batch
 - **Expected: 50-100x speedup**
 ## References
 - `piker/ui/_curve.py` - Production FlowGraphic
 - `piker/ui/_annotate.py` - GapAnnotations batch
 - `pyqtgraph/graphicsItems/BarGraphItem.py` -
  PrimitiveArray
 - `pyqtgraph/graphicsItems/ScatterPlotItem.py` -
  Fragments
 - Qt docs: QGraphicsItem caching modes
 See [examples.md](examples.md) for real-world
 optimization case studies.
 ---
 *Last updated: 2026-01-31*
 *Session: Batch gap annotation optimization*
--- a/.claude/skills/pyqtgraph-optimization/examples.md
+++ b/.claude/skills/pyqtgraph-optimization/examples.md
@ -1,84 +0,0 @@
 # PyQtGraph Optimization Examples
 Real-world optimization case studies from piker.
 ## Case Study: Gap Annotations (1285 gaps)
 ### Before: Individual `pg.ArrowItem` + `SelectRect`
 ```
 Total creation time: 6.6 seconds
 Per-item overhead: ~5ms
 Memory: 1285 ArrowItem + 1285 SelectRect objects
 ```
 Each gap was rendered as two separate
 `QGraphicsItem` instances (arrow + highlight rect),
 resulting in 2570 Qt objects.
 ### After: Single `GapAnnotations` batch renderer
 ```
 Total creation time:
  104ms (server) + 376ms (client)
 Effective per-item: ~0.08ms
 Speedup: ~36x client, ~180x server
 Memory: 1 GapAnnotations object
 ```
 All 1285 gaps rendered via:
 - One `PrimitiveArray` for all rectangles
 - One `QPainterPath` for all arrows
 - Shared pen/brush across all items
 ### Profiler Output (Client)
 ```
 > Entering markup_gaps() for 1285 gaps
  initial redraw: 0.20ms, tot:0.20
  built annotation specs: 256.48ms, tot:256.68
  batch IPC call complete: 119.26ms, tot:375.94
  final redraw: 0.07ms, tot:376.02
 < Exiting markup_gaps(), total: 376.04ms
 ```
 ### Profiler Output (Server)
 ```
 > Entering Batch annotate 1285 gaps
  `np.searchsorted()` complete!: 0.81ms, tot:0.81
  `time_to_row` creation: 98.45ms, tot:99.28
  created GapAnnotations item: 2.98ms, tot:102.26
 < Exiting Batch annotate, total: 104.15ms
 ```
 ## Positioning/Update Pattern
 For annotations that need repositioning when the
 view scrolls or zooms:
 ```python
 def reposition(self, array):
    '''
    Update positions based on new array data.
    '''
    # vectorized timestamp lookups (not linear!)
    time_to_row = self._build_lookup(array)
    # update rect array in-place
    rect_memory = self._rectarray.ndarray()
    for i, spec in enumerate(self._specs):
        row = time_to_row.get(spec['time'])
        if row:
            rect_memory[i, 0] = row['index']
            rect_memory[i, 1] = row['close']
            # ... width, height
    # trigger repaint (single call, not per-item)
    self.update()
 ```
 **Key insight:** Update the underlying memory
 arrays directly, then call `.update()` once.
 Never create/destroy Qt objects during reposition.
--- a/.claude/skills/timeseries-optimization/SKILL.md
+++ b/.claude/skills/timeseries-optimization/SKILL.md
@ -1,225 +0,0 @@
 ---
 name: timeseries-optimization
 description: >
  High-performance timeseries processing with NumPy
  and Polars for financial data. Apply when working
  with OHLCV arrays, timestamp lookups, gap
  detection, or any array/dataframe operations in
  piker.
 user-invocable: false
 ---
 # Timeseries Optimization: NumPy & Polars
 Skill for high-performance timeseries processing
 using NumPy and Polars, with focus on patterns
 common in financial/trading applications.
 ## Core Principle: Vectorization Over Iteration
 **Never write Python loops over large arrays.**
 Always look for vectorized alternatives.
 ```python
 # BAD: Python loop (slow!)
 results = []
 for i in range(len(array)):
    if array['time'][i] == target_time:
        results.append(array[i])
 # GOOD: vectorized boolean indexing (fast!)
 results = array[array['time'] == target_time]
 ```
 ## Timestamp Lookup Patterns
 The most critical optimization in piker timeseries
 code. Choose the right lookup strategy:
 ### Linear Scan (O(n)) - Avoid!
 ```python
 # BAD: O(n) scan through entire array
 for target_ts in timestamps:  # m iterations
    matches = array[array['time'] == target_ts]
    # Total: O(m * n) - catastrophic!
 ```
 **Performance:**
 - 1000 lookups x 10k array = 10M comparisons
 - Timing: ~50-100ms for 1k lookups
 ### Binary Search (O(log n)) - Good!
 ```python
 # GOOD: O(m log n) using searchsorted
 import numpy as np
 time_arr = array['time']  # extract once
 ts_array = np.array(timestamps)
 # binary search for all timestamps at once
 indices = np.searchsorted(time_arr, ts_array)
 # bounds check and exact match verification
 valid_mask = (
    (indices < len(array))
    &
    (time_arr[indices] == ts_array)
 )
 valid_indices = indices[valid_mask]
 matched_rows = array[valid_indices]
 ```
 **Requirements for `searchsorted()`:**
 - Input array MUST be sorted (ascending)
 - Works on any sortable dtype (floats, ints)
 - Returns insertion indices (not found =
  `len(array)`)
 **Performance:**
 - 1000 lookups x 10k array = ~10k comparisons
 - Timing: <1ms for 1k lookups
 - **~100-1000x faster than linear scan**
 ### Hash Table (O(1)) - Best for Repeated Lookups!
 If you'll do many lookups on same array, build
 dict once:
 ```python
 # build lookup once
 time_to_idx = {
    float(array['time'][i]): i
    for i in range(len(array))
 }
 # O(1) lookups
 for target_ts in timestamps:
    idx = time_to_idx.get(target_ts)
    if idx is not None:
        row = array[idx]
 ```
 **When to use:**
 - Many repeated lookups on same array
 - Array doesn't change between lookups
 - Can afford upfront dict building cost
 ## Performance Checklist
 When optimizing timeseries operations:
 - [ ] Is the array sorted? (enables binary search)
 - [ ] Are you doing repeated lookups?
  (build hash table)
 - [ ] Are struct fields accessed in loops?
  (extract to plain arrays)
 - [ ] Are you using boolean indexing?
  (vectorized vs loop)
 - [ ] Can operations be batched?
  (minimize round-trips)
 - [ ] Is memory being copied unnecessarily?
  (use views)
 - [ ] Are you using the right tool?
  (NumPy vs Polars)
 ## Common Bottlenecks and Fixes
 ### Bottleneck: Timestamp Lookups
 ```python
 # BEFORE: O(n*m) - 100ms for 1k lookups
 for ts in timestamps:
    matches = array[array['time'] == ts]
 # AFTER: O(m log n) - <1ms for 1k lookups
 indices = np.searchsorted(
    array['time'], timestamps,
 )
 ```
 ### Bottleneck: Dict Building from Struct Array
 ```python
 # BEFORE: 100ms for 3k rows
 result = {
    float(row['time']): {
        'index': float(row['index']),
        'close': float(row['close']),
    }
    for row in matched_rows
 }
 # AFTER: <5ms for 3k rows
 times = matched_rows['time'].astype(float)
 indices = matched_rows['index'].astype(float)
 closes = matched_rows['close'].astype(float)
 result = {
    t: {'index': idx, 'close': cls}
    for t, idx, cls in zip(
        times, indices, closes,
    )
 }
 ```
 ### Bottleneck: Repeated Field Access
 ```python
 # BEFORE: 50ms for 1k iterations
 for i, spec in enumerate(specs):
    start_row = array[
        array['time'] == spec['start_time']
    ][0]
    end_row = array[
        array['time'] == spec['end_time']
    ][0]
    process(
        start_row['index'],
        end_row['close'],
    )
 # AFTER: <5ms for 1k iterations
 # 1. Build lookup once
 time_to_row = {...}  # via searchsorted
 # 2. Extract fields to plain arrays
 indices_arr = array['index']
 closes_arr = array['close']
 # 3. Use lookup + plain array indexing
 for spec in specs:
    start_idx = time_to_row[
        spec['start_time']
    ]['array_idx']
    end_idx = time_to_row[
        spec['end_time']
    ]['array_idx']
    process(
        indices_arr[start_idx],
        closes_arr[end_idx],
    )
 ```
 ## References
 - NumPy structured arrays:
  https://numpy.org/doc/stable/user/basics.rec.html
 - `np.searchsorted`:
  https://numpy.org/doc/stable/reference/generated/numpy.searchsorted.html
 - Polars: https://pola-rs.github.io/polars/
 - `piker.tsp` - timeseries processing utilities
 - `piker.data._formatters` - OHLC array handling
 See [numpy-patterns.md](numpy-patterns.md) for
 detailed NumPy structured array patterns and
 [polars-patterns.md](polars-patterns.md) for
 Polars integration.
 ---
 *Last updated: 2026-01-31*
 *Key win: 100ms -> 5ms dict building via field
 extraction*
--- a/.claude/skills/timeseries-optimization/numpy-patterns.md
+++ b/.claude/skills/timeseries-optimization/numpy-patterns.md
@ -1,212 +0,0 @@
 # NumPy Structured Array Patterns
 Detailed patterns for working with NumPy structured
 arrays in piker's financial data processing.
 ## Piker's OHLCV Array Dtype
 ```python
 # typical piker array dtype
 dtype = [
    ('index', 'i8'),   # absolute sequence index
    ('time', 'f8'),    # unix epoch timestamp
    ('open', 'f8'),
    ('high', 'f8'),
    ('low', 'f8'),
    ('close', 'f8'),
    ('volume', 'f8'),
 ]
 arr = np.array(
    [(0, 1234.0, 100, 101, 99, 100.5, 1000)],
    dtype=dtype,
 )
 # field access
 times = arr['time']     # returns view, not copy
 closes = arr['close']
 ```
 ## Structured Array Performance Gotchas
 ### 1. Field access in loops is slow
 ```python
 # BAD: repeated struct field access per iteration
 for i, row in enumerate(arr):
    x = row['index']    # struct access!
    y = row['close']
    process(x, y)
 # GOOD: extract fields once, iterate plain arrays
 indices = arr['index']  # extract once
 closes = arr['close']
 for i in range(len(arr)):
    x = indices[i]      # plain array indexing
    y = closes[i]
    process(x, y)
 ```
 ### 2. Dict comprehensions with struct arrays
 ```python
 # SLOW: field access per row in Python loop
 time_to_row = {
    float(row['time']): {
        'index': float(row['index']),
        'close': float(row['close']),
    }
    for row in matched_rows  # struct access!
 }
 # FAST: extract to plain arrays first
 times = matched_rows['time'].astype(float)
 indices = matched_rows['index'].astype(float)
 closes = matched_rows['close'].astype(float)
 time_to_row = {
    t: {'index': idx, 'close': cls}
    for t, idx, cls in zip(
        times, indices, closes,
    )
 }
 ```
 ## Vectorized Boolean Operations
 ### Basic Filtering
 ```python
 # single condition
 recent = array[array['time'] > cutoff_time]
 # multiple conditions with &, |
 filtered = array[
    (array['time'] > start_time)
    &
    (array['time'] < end_time)
    &
    (array['volume'] > min_volume)
 ]
 # IMPORTANT: parentheses required around each!
 # (operator precedence: & binds tighter than >)
 ```
 ### Fancy Indexing
 ```python
 # boolean mask
 mask = array['close'] > array['open']  # up bars
 up_bars = array[mask]
 # integer indices
 indices = np.array([0, 5, 10, 15])
 selected = array[indices]
 # combine boolean + fancy indexing
 mask = array['volume'] > threshold
 high_vol_indices = np.where(mask)[0]
 subset = array[high_vol_indices[::2]]  # every other
 ```
 ## Common Financial Patterns
 ### Gap Detection
 ```python
 # assume sorted by time
 time_diffs = np.diff(array['time'])
 expected_step = 60.0  # 1-minute bars
 # find gaps larger than expected
 gap_mask = time_diffs > (expected_step * 1.5)
 gap_indices = np.where(gap_mask)[0]
 # get gap start/end times
 gap_starts = array['time'][gap_indices]
 gap_ends = array['time'][gap_indices + 1]
 ```
 ### Rolling Window Operations
 ```python
 # simple moving average (close)
 window = 20
 sma = np.convolve(
    array['close'],
    np.ones(window) / window,
    mode='valid',
 )
 # stride tricks for efficiency
 from numpy.lib.stride_tricks import (
    sliding_window_view,
 )
 windows = sliding_window_view(
    array['close'], window,
 )
 sma = windows.mean(axis=1)
 ```
 ### OHLC Resampling (NumPy)
 ```python
 # resample 1m bars to 5m bars
 def resample_ohlc(arr, old_step, new_step):
    n_bars = len(arr)
    factor = int(new_step / old_step)
    # truncate to multiple of factor
    n_complete = (n_bars // factor) * factor
    arr = arr[:n_complete]
    # reshape into chunks
    reshaped = arr.reshape(-1, factor)
    # aggregate OHLC
    opens = reshaped[:, 0]['open']
    highs = reshaped['high'].max(axis=1)
    lows = reshaped['low'].min(axis=1)
    closes = reshaped[:, -1]['close']
    volumes = reshaped['volume'].sum(axis=1)
    return np.rec.fromarrays(
        [opens, highs, lows, closes, volumes],
        names=[
            'open', 'high', 'low',
            'close', 'volume',
        ],
    )
 ```
 ## Memory Considerations
 ### Views vs Copies
 ```python
 # VIEW: shares memory (fast, no copy)
 times = array['time']         # field access
 subset = array[10:20]         # slicing
 reshaped = array.reshape(-1, 2)
 # COPY: new memory allocation
 filtered = array[array['time'] > cutoff]
 sorted_arr = np.sort(array)
 casted = array.astype(np.float32)
 # force copy when needed
 explicit_copy = array.copy()
 ```
 ### In-Place Operations
 ```python
 # modify in-place (no new allocation)
 array['close'] *= 1.01  # scale prices
 array['volume'][mask] = 0  # zero out rows
 # careful: compound ops may create temporaries
 array['close'] = array['close'] * 1.01  # temp!
 array['close'] *= 1.01  # true in-place
 ```
--- a/.claude/skills/timeseries-optimization/polars-patterns.md
+++ b/.claude/skills/timeseries-optimization/polars-patterns.md
@ -1,78 +0,0 @@
 # Polars Integration Patterns
 Polars usage patterns for piker's timeseries
 processing, including NumPy interop.
 ## NumPy <-> Polars Conversion
 ```python
 import polars as pl
 # numpy to polars
 df = pl.from_numpy(
    arr,
    schema=[
        'index', 'time', 'open', 'high',
        'low', 'close', 'volume',
    ],
 )
 # polars to numpy (via arrow)
 arr = df.to_numpy()
 # piker convenience
 from piker.tsp import np2pl, pl2np
 df = np2pl(arr)
 arr = pl2np(df)
 ```
 ## Polars Performance Patterns
 ### Lazy Evaluation
 ```python
 # build query lazily
 lazy_df = (
    df.lazy()
    .filter(pl.col('volume') > 1000)
    .with_columns([
        (
            pl.col('close') - pl.col('open')
        ).alias('change')
    ])
    .sort('time')
 )
 # execute once
 result = lazy_df.collect()
 ```
 ### Groupby Aggregations
 ```python
 # resample to 5-minute bars
 resampled = df.groupby_dynamic(
    index_column='time',
    every='5m',
 ).agg([
    pl.col('open').first(),
    pl.col('high').max(),
    pl.col('low').min(),
    pl.col('close').last(),
    pl.col('volume').sum(),
 ])
 ```
 ## When to Use Polars vs NumPy
 ### Use Polars when:
 - Complex queries with multiple filters/joins
 - Need SQL-like operations (groupby, window fns)
 - Working with heterogeneous column types
 - Want lazy evaluation optimization
 ### Use NumPy when:
 - Simple array operations (indexing, slicing)
 - Direct memory access needed (e.g., SHM arrays)
 - Compatibility with Qt/pyqtgraph (expects NumPy)
 - Maximum performance for numerical computation
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@ -36,26 +36,17 @@ jobs:
  testing:
    name: 'install + test-suite'
    timeout-minutes: 10
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      # elastic only
      # - name: Build DB container
      #   run: docker build -t piker:elastic dockering/elastic
      - name: Setup python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v3
        with:
          python-version: '3.10'
      # elastic only
      # - name: Install dependencies
      #   run: pip install -U .[es] -r requirements-test.txt -r requirements.txt --upgrade-strategy eager
      - name: Install dependencies
        run: pip install -U . -r requirements-test.txt -r requirements.txt --upgrade-strategy eager
--- a/.gitignore
+++ b/.gitignore
@ -98,35 +98,8 @@ ENV/
 /site
 # extra scripts dir
-# /snippets
+/snippets
 # mypy
 .mypy_cache/
 # all files under
 .git/
 # any commit-msg gen tmp files
 .claude/*_commit_*.md
 .claude/*_commit*.toml
 # nix develop --profile .nixdev
 .nixdev*
 # :Obsession .
 Session.vim
 # gitea local `.md`-files
 # TODO? would this be handy to also commit and sync with
 # wtv git hosting service tho?
 gitea/
 # ------ tina-land ------
 .vscode/settings.json
 # ------ macOS ------
 # Finder metadata
 **/.DS_Store
 # LLM conversations that should remain private
 docs/conversations/
--- a/README.rst
+++ b/README.rst
@ -1,199 +1,222 @@
 piker
 -----
-trading gear for hackers
+trading gear for hackers.
 |gh_actions|
 .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
    :target: https://actions-badge.atrox.dev/piker/pikers/goto
-``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
+``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
-real-time computational trading targeted at `hardcore Linux users
+computational trading targeted at `hardcore Linux users <comp_trader>`_ .
 <comp_trader>`_ .
-we use much bleeding edge tech including (but not limited to):
+we use as much bleeding edge tech as possible including (but not limited to):
 - latest python for glue_
- uv_ for packaging and distribution
+- trio_ for `structured concurrency`_
- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
+- tractor_ for distributed, multi-core, real-time streaming
- Qt_ for pristine low latency UIs
+- marketstore_ for historical and real-time tick data persistence and sharing
- pyqtgraph_ (which we've extended) for real-time charting and graphics
+- techtonicdb_ for L2 book storage
- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
+- Qt_ for pristine high performance UIs
- `apache arrow and parquet`_ for time-series storage
+- pyqtgraph_ for real-time charting
 - ``numpy`` and ``numba`` for `fast numerics`_
-potential projects we might integrate with soon,
+.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
-
+    :target: https://travis-ci.org/pikers/piker
 - (already prototyped in ) techtonicdb_ for L2 book storage
 .. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
 .. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
 .. _uv: https://docs.astral.sh/uv/
 .. _trio: https://github.com/python-trio/trio
 .. _tractor: https://github.com/goodboy/tractor
 .. _structured concurrency: https://trio.discourse.group/
 .. _marketstore: https://github.com/alpacahq/marketstore
 .. _techtonicdb: https://github.com/0b01/tectonicdb
 .. _Qt: https://www.qt.io/
 .. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
-.. _apache arrow and parquet: https://arrow.apache.org/faq/
+.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
 .. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
-.. _techtonicdb: https://github.com/0b01/tectonicdb
+.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
-focus and feats:
+focus and features:
 *******************
 - 100% federated: your code, your hardware, your data feeds, your broker fills.
 - zero web: low latency, native software that doesn't try to re-invent the OS
 - maximal **privacy**: prevent brokers and mms from knowing your
  planz; smack their spreads with dark volume.
 - zero clutter: modal, context oriented UIs that echew minimalism, reduce
  thought noise and encourage un-emotion.
 - first class parallelism: built from the ground up on next-gen structured concurrency
  primitives.
 - traders first: broker/exchange/asset-class agnostic
 - systems grounded: real-time financial signal processing that will
  make any queuing or DSP eng juice their shorts.
 - non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
 - data collaboration: every process and protocol is multi-host scalable.
 - fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
 fitting with these tenets, we're always open to new framework suggestions and ideas.
 building the best looking, most reliable, keyboard friendly trading
 platform is the dream; join the cause.
 install
 *******
 ``piker`` is currently under heavy pre-alpha development and as such
 should be cloned from this repo and hacked on directly.
 for a development install::
    git clone git@github.com:pikers/piker.git
    cd piker
    virtualenv env
    source ./env/bin/activate
    pip install -r requirements.txt -e .
 install for tinas
 *****************
 for windows peeps you can start by installing all the prerequisite software:
 - install git with all default settings - https://git-scm.com/download/win
 - install anaconda all default settings - https://www.anaconda.com/products/individual
 - install microsoft build tools (check the box for Desktop development for C++, you might be able to uncheck some optional downloads)  - https://visualstudio.microsoft.com/visual-cpp-build-tools/
 - install visual studio code default settings - https://code.visualstudio.com/download
 then, `crack a conda shell`_ and run the following commands::
    mkdir code # create code directory
    cd code # change directory to code
    git clone https://github.com/pikers/piker.git # downloads piker installation package from github
    cd piker # change directory to piker
    conda create -n pikonda # creates conda environment named pikonda
    conda activate pikonda # activates pikonda
    conda install -c conda-forge python-levenshtein # in case it is not already installed
    conda install pip # may already be installed
    pip # will show if pip is installed
    pip install -e . -r requirements.txt # install piker in editable mode
 test Piker to see if it is working::
    piker -b binance chart btcusdt.binance # formatting for loading a chart
    piker -b kraken -b binance chart xbtusdt.kraken
    piker -b kraken -b binance -b ib chart qqq.nasdaq.ib
    piker -b ib chart tsla.nasdaq.ib
 potential error::
    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Roaming\\piker\\brokers.toml'
 solution:
 - navigate to file directory above (may be different on your machine, location should be listed in the error code)
 - copy and paste file from 'C:\\Users\\user\\code\\data/brokers.toml' or create a blank file using notepad at the location above
 Visual Studio Code setup:
 - now that piker is installed we can set up vscode as the default terminal for running piker and editing the code
 - open Visual Studio Code
 - file --> Add Folder to Workspace --> C:\Users\user\code\piker (adds piker directory where all piker files are located)
 - file --> Save Workspace As --> save it wherever you want and call it whatever you want, this is going to be your default workspace for running and editing piker code
 - ctrl + shift + p --> start typing Python: Select Interpetter --> when the option comes up select it --> Select at the workspace level --> select the one that shows ('pikonda')
 - change the default terminal to cmd.exe instead of powershell (default)
 - now when you create a new terminal VScode should automatically activate you conda env so that piker can be run as the first command after a new terminal is created
 also, try out fancyzones as part of powertoyz for a decent tiling windows manager to manage all the cool new software you are going to be running.
 .. _conda installed: https://
 .. _C++ build toolz: https://
 .. _crack a conda shell: https://
 .. _vscode: https://
 .. link to the tina guide
 .. _setup a coolio tiled wm console: https://
 provider support
 ****************
-fitting with these tenets, we're always open to new
+for live data feeds the in-progress set of supported brokers is:
 framework/lib/service interop suggestions and ideas!
- **100% federated**:
+- IB_ via ``ib_insync``, also see our `container docs`_
-  your code, your hardware, your data feeds, your broker fills.
+- binance_ and kraken_ for crypto over their public websocket API
 - questrade_ (ish) which comes with effectively free L1
- **zero web**:
+coming soon...
  low latency as a prime objective, native UIs and modern IPC
  protocols without trying to re-invent the "OS-as-an-app"..
- **maximal privacy**:
+- webull_ via the reverse engineered public API
-  prevent brokers and mms from knowing your planz; smack their
+- yahoo via yliveticker_
  spreads with dark volume from a VPN tunnel.
- **zero clutter**:
+if you want your broker supported and they have an API let us know.
  modal, context oriented UIs that echew minimalism, reduce thought
  noise and encourage un-emotion.
- **first class parallelism**:
+.. _IB: https://interactivebrokers.github.io/tws-api/index.html
-  built from the ground up on a next-gen structured concurrency
+.. _container docs: https://github.com/pikers/piker/tree/master/dockering/ib
-  supervision sys.
+.. _questrade: https://www.questrade.com/api/documentation
-
+.. _kraken: https://www.kraken.com/features/api#public-market-data
- **traders first**:
+.. _binance: https://github.com/pikers/piker/pull/182
-  broker/exchange/venue/asset-class/money-sys agnostic
+.. _webull: https://github.com/tedchou12/webull
-
+.. _yliveticker: https://github.com/yahoofinancelive/yliveticker
- **systems grounded**:
+.. _coinbase: https://docs.pro.coinbase.com/#websocket-feed
  real-time financial signal processing (fsp) that will make any
  queuing or DSP eng juice their shorts.
 - **non-tina UX**:
  sleek, powerful keyboard driven interaction with expected use in
  tiling wms (or maybe even a DDE).
 - **data collab at scale**:
  every actor-process and protocol is multi-host aware.
 - **fight club ready**:
  zero interest in adoption by suits; no corporate friendly license,
  ever.
 building the hottest looking, fastest, most reliable, keyboard
 friendly FOSS trading platform is the dream; join the cause.
-a sane install with `uv`
+check out our charts
-************************
+********************
-bc why install with `python` when you can faster with `rust` ::
+bet you weren't expecting this from the foss::
-    uv sync
+    piker -l info -b kraken -b binance chart btcusdt.binance --pdb
    # ^ astral's docs,
    # https://docs.astral.sh/uv/concepts/projects/sync/
 include all GUIs (ex. for charting)::
    uv sync --group uis
 AND with **all** our normal hacking tools::
    uv sync --dev
 AND if you want to try WIP integrations::
    uv sync --all-groups
 Ensure you can run the root-daemon::
    uv run pikerd [-l info --pdb]
-install on nix(os)
+this runs the main chart (currently with 1m sampled OHLC) in in debug
-******************
+mode and you can practice paper trading using the following
-``NixOS`` is our core devs' distro of choice for which we offer
+micro-manual:
 a stringently defined development shell envoirment that can currently
 be applied in one of 2 ways::
-    # ONLY if running on X11
+``order_mode`` (
-    nix-shell default.nix
+    edge triggered activation by any of the following keys,
    ``mouse-click`` on y-level to submit at that price
    ):
-Or if you prefer flakes style and a modern DE::
+    - ``f``/ ``ctl-f`` to stage buy
-
+    - ``d``/ ``ctl-d`` to stage sell
-    # ONLY if also running on Wayland
+    - ``a`` to stage alert
    nix develop  # for default bash
    nix develop -c uv run xonsh  # for @goodboy's preferred sh B)
-start a chart
+``search_mode`` (
-*************
+    ``ctl-l`` or ``ctl-space`` to open,
-run a realtime OHLCV chart stand-alone::
+    ``ctl-c`` or ``ctl-space`` to close
    ) :
-    [uv run] piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
+    - begin typing to have symbol search automatically lookup
-
+      symbols from all loaded backend (broker) providers
-    # ^^^ iff you haven't activated the py-env,
+    - arrow keys and mouse click to navigate selection
-    # - https://docs.astral.sh/uv/concepts/projects/run/
+    - vi-like ``ctl-[hjkl]`` for navigation
    #
    # in order to create an explicit virt-env see,
    # - https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment
    # - https://docs.astral.sh/uv/pip/environments/
    #
    # use $UV_PROJECT_ENVIRONMENT to select any non-`.venv/`
    # as the venv sudir in the repo's root.
    # - https://docs.astral.sh/uv/reference/environment/#uv_project_environment
 this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
 overlayed on the same graph. Use of `piker` without first starting
 a daemon (`pikerd` - see below) means there is an implicit spawning of the
 multi-actor-runtime (implemented as a `tractor` app).
 For additional subsystem feats available through our chart UI see the
 various sub-readmes:
 - order control using a mouse-n-keyboard UX B)
 - cross venue market-pair (what most call "symbol") search, select, overlay Bo
 - financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
 - src-asset derivatives scan for anal, like the infamous "max pain" XO
-spawn a daemon standalone
+you can also configure your position allocation limits from the
-*************************
+sidepane.
-we call the root actor-process the ``pikerd``. it can be (and is
+
-recommended normally to be) started separately from the ``piker
+
-chart`` program::
+run in distributed mode
 ***********************
 start the service manager and data feed daemon in the background and
 connect to it::
    pikerd -l info --pdb
 the daemon does nothing until a ``piker``-client (like ``piker
 chart``) connects and requests some particular sub-system. for
 a connecting chart ``pikerd`` will spawn and manage at least,
- a data-feed daemon: ``datad`` which does all the work of comms with
+connect your chart::
  the backend provider (in this case the ``binance`` cex).
 - a paper-trading engine instance, ``paperboi.binance``, (if no live
  account has been configured) which allows for auto/manual order
  control against the live quote stream.
-*using* an actor-service (aka micro-daemon) manager which dynamically
+    piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
 supervises various sub-subsystems-as-services throughout the ``piker``
 runtime-stack.
 now you can (implicitly) connect your chart::
-    piker chart btcusdt.spot.binance
+enjoy persistent real-time data feeds tied to daemon lifetime. the next
-
+time you spawn a chart it will load much faster since the data feed has
-since ``pikerd`` was started separately you can now enjoy a persistent
+been cached and is now always running live in the background until you
-real-time data stream tied to the daemon-tree's lifetime. i.e. the next
+kill ``pikerd``.
 time you spawn a chart it will obviously not only load much faster
 (since the underlying ``datad.binance`` is left running with its
 in-memory IPC data structures) but also the data-feed and any order
 mgmt states should be persistent until you finally cancel ``pikerd``.
 if anyone asks you what this project is about
 *********************************************
-you don't talk about it; just use it.
+you don't talk about it.
 how do i get involved?
@ -203,15 +226,6 @@ enter the matrix.
 how come there ain't that many docs
 ***********************************
-i mean we want/need them but building the core right has been higher
+suck it up, learn the code; no one is trying to sell you on anything.
-prio then marketting (and likely will stay that way Bp).
+also, we need lotsa help so if you want to start somewhere and can't
-
+necessarily write serious code, this might be the place for you!
 soo, suck it up bc,
 - no one is trying to sell you on anything
 - learning the code base is prolly way more valuable
 - the UI/UXs are intended to be "intuitive" for any hacker..
 we obviously need tonz help so if you want to start somewhere and
 can't necessarily write "advanced" concurrent python/rust code, this
 helping document literally anything might be the place for you!
--- a/ai/README.md
+++ b/ai/README.md
@ -1,50 +0,0 @@
 # AI Tooling Integrations
 Documentation and usage guides for AI-assisted
 development tools integrated with this repo.
 Each subdirectory corresponds to a specific AI tool
 or frontend and contains usage docs for the
 custom skills/prompts/workflows configured for it.
 Originally introduced in
 [PR #69](https://www.pikers.dev/pikers/piker/pulls/69);
 track new integration ideas and proposals in
 [issue #79](https://www.pikers.dev/pikers/piker/issues/79).
 ## Integrations
 | Tool | Directory | Status |
 |------|-----------|--------|
 | [Claude Code](https://github.com/anthropics/claude-code) | [`claude-code/`](claude-code/) | active |
 ## Adding a New Integration
 Create a subdirectory named after the tool (use
 lowercase + hyphens), then add:
 1. A `README.md` covering setup, available
   skills/commands, and usage examples
 2. Any tool-specific config or prompt files
 ```
 ai/
 ├── README.md                  # <- you are here
 ├── claude-code/
 │   └── README.md
 ├── opencode/                  # future
 │   └── README.md
 └── <your-tool>/
    └── README.md
 ```
 ## Conventions
 - Skill/command names use **hyphen-case**
  (`commit-msg`, not `commit_msg`)
 - Each integration doc should describe **what**
  the skill does, **how** to invoke it, and any
  **output** artifacts it produces
 - Keep docs concise; link to the actual skill
  source files (under `.claude/skills/`, etc.)
  rather than duplicating content
--- a/ai/claude-code/README.md
+++ b/ai/claude-code/README.md
@ -1,183 +0,0 @@
 # Claude Code Integration
 [Claude Code](https://github.com/anthropics/claude-code)
 skills and workflows for piker development.
 ## Skills
 | Skill | Invocable | Description |
 |-------|-----------|-------------|
 | [`commit-msg`](#commit-msg) | `/commit-msg` | Generate piker-style commit messages |
 | `piker-profiling` | auto | `Profiler` API patterns for perf work |
 | `piker-slang` | auto | Communication style + slang guide |
 | `pyqtgraph-optimization` | auto | Batch rendering patterns |
 | `timeseries-optimization` | auto | NumPy/Polars perf patterns |
 Skills marked **auto** are background knowledge
 applied automatically when Claude detects relevance.
 Only `commit-msg` is user-invoked via slash command.
 Skill source files live under
 `.claude/skills/<skill-name>/SKILL.md`.
 ---
 ## `/commit-msg`
 Generate piker-style git commit messages trained on
 500+ commits from the repo history.
 ### Quick Start
 ```
 # basic - analyzes staged diff automatically
 /commit-msg
 # with scope hint
 /commit-msg .ib.feed: fix bar trimming
 # with description context
 /commit-msg refactor position tracking
 ```
 ### What It Does
 1. **Reads staged changes** via dynamic context
   injection (`git diff --staged --stat`)
 2. **Reads recent commits** for style reference
   (`git log --oneline -10`)
 3. **Generates** a commit message following
   piker conventions (verb choice, backtick refs,
   colon prefixes, section markers, etc.)
 4. **Writes** the message to two files:
   - `.claude/<timestamp>_<hash>_commit_msg.md`
   - `.claude/git_commit_msg_LATEST.md`
     (overwritten each time)
 ### Arguments
 The optional argument after `/commit-msg` is
 passed as `$ARGUMENTS` and used as scope or
 description context. Examples:
 | Invocation | Effect |
 |------------|--------|
 | `/commit-msg` | Infer scope from diff |
 | `/commit-msg .ib.feed` | Use `.ib.feed:` prefix |
 | `/commit-msg fix the null seg crash` | Use as description hint |
 ### Output Format
 **Subject line:**
 - ~50 chars target, 67 max
 - Present tense verb (Add, Drop, Fix, Factor..)
 - Backtick-wrapped code refs
 - Optional module prefix (`.ib.feed: ...`)
 **Body** (when needed):
 - 67 char line max
 - Section markers: `Also,`, `Deats,`, `Further,`
 - `-` bullet lists for multiple changes
 - Piker abbreviations (`msg`, `mod`, `impl`,
  `deps`, `bc`, `obvi`, `prolly`..)
 **Footer** (always):
 ```
 (this patch was generated in some part by
 [`claude-code`][claude-code-gh])
 [claude-code-gh]: https://github.com/anthropics/claude-code
 ```
 ### Output Files
 After generation, the commit message is written to:
 ```
 .claude/
 ├── <timestamp>_<hash>_commit_msg.md   # archived
 └── git_commit_msg_LATEST.md           # latest
 ```
 Where `<timestamp>` is ISO-8601 with seconds and
 `<hash>` is the first 7 chars of the current
 `HEAD` commit.
 Use the latest file to feed into `git commit`:
 ```bash
 git commit -F .claude/git_commit_msg_LATEST.md
 ```
 Or review/edit before committing:
 ```bash
 cat .claude/git_commit_msg_LATEST.md
 # edit if needed, then:
 git commit -F .claude/git_commit_msg_LATEST.md
 ```
 ### Examples
 **Simple one-liner output:**
 ```
 Add `MktPair.fqme` property for symbol resolution
 ```
 **Multi-file change output:**
 ```
 Factor `.claude/skills/` into proper subdirs
 Deats,
 - `commit_msg/` -> `commit-msg/` w/ enhanced
  frontmatter
 - all background skills set `user-invocable: false`
 - content split into supporting files
 (this patch was generated in some part by
 [`claude-code`][claude-code-gh])
 [claude-code-gh]: https://github.com/anthropics/claude-code
 ```
 ### Frontmatter Reference
 The skill's `SKILL.md` uses these Claude Code
 frontmatter fields:
 ```yaml
 ---
 name: commit-msg
 description: >
  Generate piker-style git commit messages...
 argument-hint: "[optional-scope-or-description]"
 disable-model-invocation: true
 allowed-tools:
  - Bash(git *)
  - Read
  - Grep
  - Glob
  - Write
 ---
 ```
 | Field | Purpose |
 |-------|---------|
 | `argument-hint` | Shows hint in autocomplete |
 | `disable-model-invocation` | Only user can trigger via `/commit-msg` |
 | `allowed-tools` | Tools the skill can use |
 ### Dynamic Context
 The skill injects live data at invocation time
 via `!`backtick`` syntax in the `SKILL.md`:
 ```markdown
 ## Current staged changes
 !`git diff --staged --stat`
 ## Recent commit style reference
 !`git log --oneline -10`
 ```
 This means the staged diff stats and recent log
 are always fresh when the skill runs -- no stale
 context.
--- a/config/brokers.toml
+++ b/config/brokers.toml
@ -1,108 +1,57 @@
-# ---- CEXY ----
+[questrade]
-
+refresh_token = ""
-[binance]
+access_token = ""
-accounts.paper = 'paper'
+api_server = "https://api06.iq.questrade.com/"
-
+expires_in = 1800
-accounts.usdtm = 'futes'
+token_type = "Bearer"
-futes.use_testnet = false
+expires_at = 1616095326.355846
 futes.api_key = ''
 futes.api_secret = ''
 accounts.spot = 'spot'
 spot.use_testnet = false
 spot.api_key = ''
 spot.api_secret = ''
 # ------ binance ------
 [deribit]
 # std assets
 key_id = ''
 key_secret = ''
 # options
 accounts.option = 'option'
 option.use_testnet = false
 option.key_id = ''
 option.key_secret = ''
 # aux logging from `cryptofeed`
 option.log.filename = 'cryptofeed.log'
 option.log.level = 'DEBUG'
 option.log.disabled = true
 # ------ deribit ------
 [kraken]
-key_descr = ''
+key_descr = "api_0"
-api_key = ''
+api_key = ""
-secret = ''
+secret = ""
 # ------ kraken ------
 [kucoin]
 key_id = ''
 key_secret = ''
 key_passphrase = ''
 # ------ kucoin ------
 # -- BROKERZ ---
 [questrade]
 refresh_token = ''
 access_token = ''
 api_server = 'https://api06.iq.questrade.com/'
 expires_in = 1800
 token_type = 'Bearer'
 expires_at = 1616095326.355846
 # ------ questrade ------
 [ib]
 # define the (set of) host-port socketaddrs that
 # brokerd.ib will scan to connect to an API endpoint
 # (ib-gw or ib-tws listening instances)
 hosts = [
-    '127.0.0.1',
+    "127.0.0.1",
 ]
 # XXX: the order in which ports will be scanned
 # (by the `brokerd` daemon-actor)
 # is determined # by the line order here.
 # TODO: when we eventually spawn gateways in our
 # container, we can just dynamically allocate these
 # using IBC.
 ports = [
    4002,  # gw
    7497,  # tws
 ]
-# When API endpoints are being scanned durin startup, the order
+# XXX: for a paper account the flex web query service
-# of user-defined-account "names" (as defined below) here
+# is not supported so you have to manually download
-# determines which py-client connection is given priority to be
+# and XML report and put it in a location that can be
-# used for data-feed-requests by according to whichever client
+# accessed by the ``brokerd.ib`` backend code for parsing.
-# connected to an API endpoing which reported the equivalent
+flex_token = '666666666666666666666666'
-# account number for that name.
+flex_trades_query_id = '666666'  # live account
 # when clients are being scanned this determines
 # which clients are preferred to be used for data
 # feeds based on the order of account names, if
 # detected as active on an API client.
 prefer_data_account = [
    'paper',
    'margin',
    'ira',
 ]
 # For long-term trades txn (transaction) history
 # processing (i.e your txn ledger with IB) you can
 # (automatically for live accounts) query the FLEX
 # report system for past history.
 #
 # (For paper accounts the web query service
 # is not supported so you have to manually download
 # an XML report and put it in a location that can be
 # accessed by our `brokerd.ib` backend code for parsing).
 #
 flex_token = ''
 flex_trades_query_id = ''  # live account
 # define "aliases" (names) for each account number
 # such that the names can be reffed and logged throughout
 # `piker.accounting` subsys and more easily
 # referred to  by the user.
 #
 # These keys will be the set exposed through the order-mode
 # account-selection UI so that numbers are never shown.
 [ib.accounts]
-paper = 'DU0000000'  # <- literal account #
+# the order in which accounts will be selectable
-margin = 'U0000000'
+# in the order mode UI (if found via clients during
-ira = 'U0000000'
+# API-app scanning)when a new symbol is loaded.
-# ------ ib ------
+paper = "XX0000000"
 margin = "X0000000"
 ira = "X0000000"
 [deribit]
 key_id = 'XXXXXXXX'
 key_secret = 'Xx_XxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXx'
--- a/config/conf.toml
+++ b/config/conf.toml
@ -1,14 +0,0 @@
 [network]
 pikerd = [
  '/ipv4/127.0.0.1/tcp/6116',  # std localhost daemon-actor tree
  # '/uds/6116',  # TODO std uds socket file
 ]
 [ui]
 # set custom font + size which will scale entire UI
 # font_size = 16
 # font_name = 'Monospaced'
 # colorscheme = 'default'  # UNUSED
 # graphics.update_throttle = 60  # Hz  # TODO
--- a/default.nix
+++ b/default.nix
@ -1,135 +0,0 @@
 with (import <nixpkgs> {});
 let
  glibStorePath = lib.getLib glib;
  zlibStorePath = lib.getLib zlib;
  zstdStorePath = lib.getLib zstd;
  dbusStorePath = lib.getLib dbus;
  libGLStorePath = lib.getLib libGL;
  freetypeStorePath = lib.getLib freetype;
  qt6baseStorePath = lib.getLib qt6.qtbase;
  fontconfigStorePath = lib.getLib fontconfig;
  libxkbcommonStorePath = lib.getLib libxkbcommon;
  xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
  pypkgs = python313Packages;
  qtpyStorePath = lib.getLib pypkgs.qtpy;
  pyqt6StorePath = lib.getLib pypkgs.pyqt6;
  pyqt6SipStorePath = lib.getLib pypkgs.pyqt6-sip;
  rapidfuzzStorePath = lib.getLib pypkgs.rapidfuzz;
  qdarkstyleStorePath = lib.getLib pypkgs.qdarkstyle;
  xorgLibX11StorePath = lib.getLib xorg.libX11;
  xorgLibxcbStorePath = lib.getLib xorg.libxcb;
  xorgxcbutilwmStorePath = lib.getLib xorg.xcbutilwm;
  xorgxcbutilimageStorePath = lib.getLib xorg.xcbutilimage;
  xorgxcbutilerrorsStorePath = lib.getLib xorg.xcbutilerrors;
  xorgxcbutilkeysymsStorePath = lib.getLib xorg.xcbutilkeysyms;
  xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
 in
 stdenv.mkDerivation {
  name = "piker-qt6-uv";
  buildInputs = [
    # System requirements.
    glib
    zlib
    dbus
    zstd
    libGL
    freetype
    qt6.qtbase
    libgcc.lib
    fontconfig
    libxkbcommon
    # Xorg requirements
    xcb-util-cursor
    xorg.libxcb
    xorg.libX11
    xorg.xcbutilwm
    xorg.xcbutilimage
    xorg.xcbutilerrors
    xorg.xcbutilkeysyms
    xorg.xcbutilrenderutil
    # Python requirements.
    python313
    uv
    pypkgs.qdarkstyle
    pypkgs.rapidfuzz
    pypkgs.pyqt6
    pypkgs.qtpy
  ];
  src = null;
  shellHook = ''
    set -e
    # Set the Qt plugin path
    # export QT_DEBUG_PLUGINS=1
    QTBASE_PATH="${qt6baseStorePath}/lib"
    QT_PLUGIN_PATH="$QTBASE_PATH/qt-6/plugins"
    QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
    LIB_GCC_PATH="${libgcc.lib}/lib"
    GLIB_PATH="${glibStorePath}/lib"
    ZSTD_PATH="${zstdStorePath}/lib"
    ZLIB_PATH="${zlibStorePath}/lib"
    DBUS_PATH="${dbusStorePath}/lib"
    LIBGL_PATH="${libGLStorePath}/lib"
    FREETYPE_PATH="${freetypeStorePath}/lib"
    FONTCONFIG_PATH="${fontconfigStorePath}/lib"
    LIB_XKB_COMMON_PATH="${libxkbcommonStorePath}/lib"
    XCB_UTIL_CURSOR_PATH="${xcbutilcursorStorePath}/lib"
    XORG_LIB_X11_PATH="${xorgLibX11StorePath}/lib"
    XORG_LIB_XCB_PATH="${xorgLibxcbStorePath}/lib"
    XORG_XCB_UTIL_IMAGE_PATH="${xorgxcbutilimageStorePath}/lib"
    XORG_XCB_UTIL_WM_PATH="${xorgxcbutilwmStorePath}/lib"
    XORG_XCB_UTIL_RENDER_UTIL_PATH="${xorgxcbutilrenderutilStorePath}/lib"
    XORG_XCB_UTIL_KEYSYMS_PATH="${xorgxcbutilkeysymsStorePath}/lib"
    XORG_XCB_UTIL_ERRORS_PATH="${xorgxcbutilerrorsStorePath}/lib"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_GCC_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$DBUS_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GLIB_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZLIB_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZSTD_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBGL_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FONTCONFIG_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FREETYPE_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_XKB_COMMON_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XCB_UTIL_CURSOR_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_X11_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_XCB_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_IMAGE_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_WM_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_RENDER_UTIL_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_KEYSYMS_PATH"
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_ERRORS_PATH"
    export LD_LIBRARY_PATH
    RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.13/site-packages"
    QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.13/site-packages"
    QTPY_PATH="${qtpyStorePath}/lib/python3.13/site-packages"
    PYQT6_PATH="${pyqt6StorePath}/lib/python3.13/site-packages"
    PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.13/site-packages"
    PATCH="$PATCH:$RPDFUZZ_PATH"
    PATCH="$PATCH:$QDRKSTYLE_PATH"
    PATCH="$PATCH:$QTPY_PATH"
    PATCH="$PATCH:$PYQT6_PATH"
    PATCH="$PATCH:$PYQT6_SIP_PATH"
    export PATCH
    # install all dev and extras
    uv sync --dev --all-extras
  '';
 }
--- a/develop.nix
+++ b/develop.nix
@ -1,47 +0,0 @@
 with (import <nixpkgs> {});
 stdenv.mkDerivation {
  name = "poetry-env";
  buildInputs = [
    # System requirements.
    readline
    # TODO: hacky non-poetry install stuff we need to get rid of!!
    poetry
    # virtualenv
    # setuptools
    # pip
    # Python requirements (enough to get a virtualenv going).
    python311Full
    # obviously, and see below for hacked linking
    python311Packages.pyqt5
    python311Packages.pyqt5_sip
    # python311Packages.qtpy
    # numerics deps
    python311Packages.levenshtein
    python311Packages.fastparquet
    python311Packages.polars
  ];
  # environment.sessionVariables = {
  #   LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
  # };
  src = null;
  shellHook = ''
    # Allow the use of wheels.
    SOURCE_DATE_EPOCH=$(date +%s)
    # Augment the dynamic linker path
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
    export QT_QPA_PLATFORM_PLUGIN_PATH="${qt5.qtbase.bin}/lib/qt-${qt5.qtbase.version}/plugins";
    if [ ! -d ".venv" ]; then
        poetry install --with uis
    fi
    poetry shell
  '';
 }
--- a/dockering/elastic/Dockerfile
+++ b/dockering/elastic/Dockerfile
@ -1,11 +0,0 @@
 FROM elasticsearch:7.17.4
 ENV ES_JAVA_OPTS "-Xms2g -Xmx2g"
 ENV ELASTIC_USERNAME "elastic"
 ENV ELASTIC_PASSWORD "password"
 COPY elasticsearch.yml /usr/share/elasticsearch/config/
 RUN printf "password" | ./bin/elasticsearch-keystore add -f -x "bootstrap.password"
 EXPOSE 19200
--- a/dockering/elastic/elasticsearch.yml
+++ b/dockering/elastic/elasticsearch.yml
@ -1,5 +0,0 @@
 network.host: 0.0.0.0
 http.port: 19200
 discovery.type: single-node
--- a/dockering/ib/README.rst
+++ b/dockering/ib/README.rst
@ -1,138 +1,30 @@
 running ``ib`` gateway in ``docker``
 ------------------------------------
-We have a config based on a well maintained community
+We have a config based on the (now defunct)
-image from `@gnzsnz`:
+image from "waytrade":
-https://github.com/gnzsnz/ib-gateway-docker
+https://github.com/waytrade/ib-gateway-docker
-
+To startup this image with our custom settings
-To startup this image simply run the command::
+simply run the command::
    docker compose up
-(For further usage^ see the official `docker-compose`_ docs)
+And you should have the following socket-available services:
 - ``x11vnc1@127.0.0.1:3003``
 - ``ib-gw@127.0.0.1:4002``
-And you should have the following socket-available services by
+You can attach to the container via a VNC client
-default:
+without password auth.
- ``x11vnc1 @ 127.0.0.1:5900``
+SECURITY STUFF!?!?!
- ``ib-gw @ 127.0.0.1:4002``
+-------------------
-
+Though "``ib``" claims they host filter connections outside
-You can now attach to the container via a VNC client with password-auth;
+localhost (aka ``127.0.0.1``) it's probably better if you filter
-here is an example using ``vncclient`` on ``linux``::
+the socket at the OS level using a stateless firewall rule::
    vncviewer localhost:5900
 now enter the pw (password) you set via an (see second code blob)
 `.env file`_ or pw-file according to the `credentials section`_.
 If you want to change away from their default config see the example
 `docker-compose.yml`-config issue and config-section of the readme,
  - https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
  - https://github.com/gnzsnz/ib-gateway-docker/discussions/103
 .. _.env file: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#how-to-use-it
 .. _docker-compose: https://docs.docker.com/compose/
 .. _credentials section: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#credentials
 Connecting to the API from `piker`
 ---------------------------------
 In order to expose the container's API endpoint to the
 `brokerd/datad/ib` actor, we need to add a section to the user's
 `brokers.toml` config (note the below is similar to the repo-shipped
 template file),
 .. code:: toml
    [ib]
    # define the (set of) host-port socketaddrs that
    # brokerd.ib will scan to connect to an API endpoint
    # (ib-gw or ib-tws listening instances)
    hosts = [
        '127.0.0.1',
    ]
    ports = [
        4002,  # gw
        7497,  # tws
    ]
    # When API endpoints are being scanned durin startup, the order
    # of user-defined-account "names" (as defined below) here
    # determines which py-client connection is given priority to be
    # used for data-feed-requests by according to whichever client
    # connected to an API endpoing which reported the equivalent
    # account number for that name.
    prefer_data_account = [
        'paper',
        'margin',
        'ira',
    ]
    # define "aliases" (names) for each account number
    # such that the names can be reffed and logged throughout
    # `piker.accounting` subsys and more easily
    # referred to  by the user.
    #
    # These keys will be the set exposed through the order-mode
    # account-selection UI so that numbers are never shown.
    [ib.accounts]
    paper = 'XX0000000'
    margin = 'X0000000'
    ira = 'X0000000'
 the broker daemon can also connect to the container's VNC server for
 added functionalies including,
 - viewing the API endpoint program's GUI for manual interventions,
 - workarounds for historical data throttling using hotkey hacks,
 Add a further section to `brokers.toml` which maps each API-ep's
 port to a table of VNC server connection info like,
 .. code:: toml
    [ib.vnc_addrs]
    4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
 The `pw = 'doggy'` here ^ should the same value as the particular
 container instances `.env` file setting (when it was run),
 .. code:: ini
    VNC_SERVER_PASSWORD='doggy'
 IF you also want to run ``TWS``
 -------------------------------
 You can also run it containerized,
 https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#using-tws
 SECURITY stuff (advanced, only if you're paranoid)
 --------------------------------------------------
 First and foremost if doing a "distributed" container setup where you
 run the ``ib-gw`` docker container and your connecting API client
 (likely ``ib_async`` from python) on **different hosts** be sure to
 read the `security considerations`_ section!
 And for a further (somewhat paranoid) perspective from
 a long-time-ago serious devops eng..
 Though "``ib``" claims they filter remote host connections outside
 ``localhost`` (aka ``127.0.0.1`` on ipv4) it's prolly justified if
 you'd like to filter the socket at the *OS level* using a stateless
 firewall rule::
    ip rule add not unicast iif lo to 0.0.0.0/0 dport 4002
-
+We will soon have this baked into our own custom image but for
-We will soon have this either baked into our own custom derivative
+now you'll have to do it urself dawgy.
 image (or patched into the current upstream one after further testin)
 but for now you'll have to do it urself, diggity dawg.
 .. _security considerations: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#security-considerations
--- a/dockering/ib/docker-compose.yml
+++ b/dockering/ib/docker-compose.yml
@ -1,32 +1,13 @@
-# a community maintained IB API container!
+# rework from the original @
-#
+# https://github.com/waytrade/ib-gateway-docker/blob/master/docker-compose.yml
-# https://github.com/gnzsnz/ib-gateway-docker
+version: "3.5"
 #
 # For piker we (currently) include some minor deviations
 # for some config files in the `volumes` section.
 #
 # See full configuration settings @
 # - https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
 # - https://github.com/gnzsnz/ib-gateway-docker/discussions/103
 services:
  ib_gw_paper:
    # apparently java is a mega cukc:
    # https://stackoverflow.com/a/56895801
    # https://bugs.openjdk.org/browse/JDK-8150460
    ulimits:
      # nproc: 65535
      nproc: 6000
      nofile:
        soft: 2000
        hard: 3000
    # other image tags available:
    # https://github.com/waytrade/ib-gateway-docker#supported-tags
-    # image: waytrade/ib-gateway:1012.2i
+    # image: waytrade/ib-gateway:981.3j
-    image: ghcr.io/gnzsnz/ib-gateway:latest
+    image: waytrade/ib-gateway:1012.2i
    restart: 'no'  # restart on boot whenev there's a crash or user clicsk
    network_mode: 'host'
@ -55,22 +36,16 @@ services:
        target: /root/scripts/run_x11_vnc.sh
        read_only: true
-    # NOTE: an alt method to fill these out is to
+    # NOTE:to fill these out, define an `.env` file in the same dir as
-    # define an `.env` file in the same dir as
+    # this compose file which looks something like:
-    # this compose file.
+    # TWS_USERID='myuser'
    # TWS_PASSWORD='guest'
    environment:
      TWS_USERID: ${TWS_USERID}
      # TWS_USERID: 'myuser'
      TWS_PASSWORD: ${TWS_PASSWORD}
-      # TWS_PASSWORD: 'guest'
+      TRADING_MODE: 'paper'
-      TRADING_MODE: ${TRADING_MODE}
+      VNC_SERVER_PASSWORD: 'doggy'
-      # TRADING_MODE: 'paper'
+      VNC_SERVER_PORT: '3003'
      VNC_SERVER_PASSWORD: ${VNC_SERVER_PASSWORD}
      # VNC_SERVER_PASSWORD: 'doggy'
      # TODO, see if we can get this supported like it
      # was on the old `waytrade` image?
      # VNC_SERVER_PORT: '3003'
    # ports:
    #   - target: 4002
@ -87,9 +62,6 @@ services:
      # - "127.0.0.1:4002:4002"
      # - "127.0.0.1:5900:5900"
  # TODO, a masked but working example of dual paper + live
  # ib-gw instances running in a single app run!
  #
  # ib_gw_live:
  #   image: waytrade/ib-gateway:1012.2i
  #   restart: no
--- a/dockering/ib/ibc.ini
+++ b/dockering/ib/ibc.ini
@ -117,57 +117,9 @@ SecondFactorDevice=
 # If you use the IBKR Mobile app for second factor authentication,
 # and you fail to complete the process before the time limit imposed
-# by IBKR, this setting tells IBC whether to automatically restart
+# by IBKR, you can use this setting to tell IBC to exit: arrangements
-# the login sequence, giving you another opportunity to complete
+# can then be made to automatically restart IBC in order to initiate
-# second factor authentication. 
+# the login sequence afresh. Otherwise, manual intervention at TWS's
 #
 # Permitted values are 'yes' and 'no'.
 #
 # If this setting is not present or has no value, then the value
 # of the deprecated ExitAfterSecondFactorAuthenticationTimeout is
 # used instead. If this also has no value, then this setting defaults
 # to 'no'.
 #
 # NB: you must be using IBC v3.14.0 or later to use this setting:
 # earlier versions ignore it.
 ReloginAfterSecondFactorAuthenticationTimeout=
 # This setting is only relevant if
 # ReloginAfterSecondFactorAuthenticationTimeout is set to 'yes',
 # or if ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
 #
 # It controls how long (in seconds) IBC waits for login to complete
 # after the user acknowledges the second factor authentication
 # alert at the IBKR Mobile app. If login has not completed after
 # this time, IBC terminates.
 # The default value is 60.
 SecondFactorAuthenticationExitInterval=
 # This setting specifies the timeout for second factor authentication
 # imposed by IB. The value is in seconds. You should not change this
 # setting unless you have reason to believe that IB has changed the
 # timeout. The default value is 180.
 SecondFactorAuthenticationTimeout=180
 # DEPRECATED SETTING
 # ------------------
 #
 # ExitAfterSecondFactorAuthenticationTimeout - THIS SETTING WILL BE
 # REMOVED IN A FUTURE RELEASE. For IBC version 3.14.0 and later, see
 # the notes for ReloginAfterSecondFactorAuthenticationTimeout above.
 #
 # For IBC versions earlier than 3.14.0: If you use the IBKR Mobile
 # app for second factor authentication, and you fail to complete the
 # process before the time limit imposed by IBKR, you can use this
 # setting to tell IBC to exit: arrangements can then be made to
 # automatically restart IBC in order to initiate the login sequence
 # afresh. Otherwise, manual intervention at TWS's
 # Second Factor Authentication dialog is needed to complete the
 # login.
 #
@ -180,18 +132,29 @@ SecondFactorAuthenticationTimeout=180
 ExitAfterSecondFactorAuthenticationTimeout=no
 # This setting is only relevant if
 # ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
 #
 # It controls how long (in seconds) IBC waits for login to complete
 # after the user acknowledges the second factor authentication
 # alert at the IBKR Mobile app. If login has not completed after
 # this time, IBC terminates.
 # The default value is 40.
 SecondFactorAuthenticationExitInterval=
 # Trading Mode
 # ------------
 #
-# This indicates whether the live account or the paper trading
+# TWS 955 introduced a new Trading Mode combo box on its login
-# account corresponding to the supplied credentials is to be used.
+# dialog. This indicates whether the live account or the paper
-# The allowed values are 'live' (the default) and 'paper'.
+# trading account corresponding to the supplied credentials is
-#
+# to be used. The allowed values are 'live' (the default) and
-# If this is set to 'live', then the credentials for the live
+# 'paper'. For earlier versions of TWS this setting has no
-# account must be supplied. If it is set to 'paper', then either
+# effect.
 # the live or the paper-trading credentials may be supplied.
-TradingMode=paper
+TradingMode=
 # Paper-trading Account Warning
@ -225,7 +188,7 @@ AcceptNonBrokerageAccountWarning=yes
 #
 # The default value is 60.
-LoginDialogDisplayTimeout=60
+LoginDialogDisplayTimeout=20
@ -254,15 +217,7 @@ LoginDialogDisplayTimeout=60
 # but they are acceptable.
 #
 # The default is the current working directory when IBC is
-# started, unless the TWS_SETTINGS_PATH setting in the relevant
+# started.
 # start script is set.
 #
 # If both this setting and TWS_SETTINGS_PATH are set, then this
 # setting takes priority. Note that if they have different values,
 # auto-restart will not work.
 #
 # NB: this setting is now DEPRECATED. You should use the
 # TWS_SETTINGS_PATH setting in the relevant start script.
 IbDir=/root/Jts
@ -329,32 +284,15 @@ ExistingSessionDetectedAction=primary
 # Override TWS API Port Number
 # ----------------------------
 #
-# If OverrideTwsApiPort is set to an integer, IBC changes the 
+# If OverrideTwsApiPort is set to an integer, IBC changes the
-# 'Socket port' in TWS's API configuration to that number shortly 
+# 'Socket port' in TWS's API configuration to that number shortly
-# after startup (but note that for the FIX Gateway, this setting is
+# after startup. Leaving the setting blank will make no change to
-# actually stored in jts.ini rather than the Gateway's settings
+# the current setting. This setting is only intended for use in
-# file). Leaving the setting blank will make no change to 
+# certain specialized situations where the port number needs to
 # the current setting. This setting is only intended for use in 
 # certain specialized situations where the port number needs to 
 # be set dynamically at run-time, and for the FIX Gateway: most
 # non-FIX users will never need it, so don't use it unless you know
 # you need it.
 OverrideTwsApiPort=4000
 # Override TWS Master Client ID
 # -----------------------------
 #
 # If OverrideTwsMasterClientID is set to an integer, IBC changes the
 # 'Master Client ID' value in TWS's API configuration to that 
 # value shortly after startup. Leaving the setting blank will make
 # no change to the current setting. This setting is only intended 
 # for use in certain specialized situations where the value needs to
 # be set dynamically at run-time: most users will never need it,
 # so don't use it unless you know you need it.
-OverrideTwsMasterClientID=
+; OverrideTwsApiPort=4002
 # Read-only Login
@ -364,13 +302,11 @@ OverrideTwsMasterClientID=
 # account security programme, the user will not be asked to perform
 # the second factor authentication action, and login to TWS will
 # occur automatically in read-only mode: in this mode, placing or
-# managing orders is not allowed. 
+# managing orders is not allowed. If set to 'no', and the user is
-#
+# enrolled in IB's account security programme, the user must perform
-# If set to 'no', and the user is enrolled in IB's account security
+# the relevant second factor authentication action to complete the
-# programme, the second factor authentication process is handled
+# login. 
-# according to the Second Factor Authentication Settings described
+
 # elsewhere in this file.
 #
 # If the user is not enrolled in IB's account security programme,
 # this setting is ignored. The default is 'no'.
@ -390,44 +326,7 @@ ReadOnlyLogin=no
 # set the relevant checkbox (this only needs to be done once) and
 # not provide a value for this setting.
-ReadOnlyApi=
+ReadOnlyApi=no
 # API Precautions
 # ---------------
 # 
 # These settings relate to the corresponding 'Precautions' checkboxes in the
 # API section of the Global Configuration dialog.
 #
 # For all of these, the accepted values are:
 # - 'yes' sets the checkbox
 # - 'no' clears the checkbox
 # - if not set, the existing TWS/Gateway configuration is unchanged
 #
 # NB: thess settings are really only supplied for the benefit of new TWS
 # or Gateway instances that are being automatically installed and
 # started without user intervention, or where user settings are not preserved
 # between sessions (eg some Docker containers). Where a user is involved, they
 # should use the Global Configuration to set the relevant checkboxes and not
 # provide values for these settings.
 BypassOrderPrecautions=
 BypassBondWarning=
 BypassNegativeYieldToWorstConfirmation=
 BypassCalledBondWarning=
 BypassSameActionPairTradeWarning=
 BypassPriceBasedVolatilityRiskWarning=
 BypassUSStocksMarketDataInSharesWarning=
 BypassRedirectOrderWarning=
 BypassNoOverfillProtectionPrecaution=
 # Market data size for US stocks - lots or shares
@ -482,145 +381,54 @@ AcceptBidAskLastSizeDisplayUpdateNotification=accept
 SendMarketDataInLotsForUSstocks=
 # Trusted API Client IPs
 # ----------------------
 #
 # NB: THIS SETTING IS ONLY RELEVANT FOR THE GATEWAY, AND ONLY WHEN FIX=yes.
 # In all other cases it is ignored.
 #
 # This is a list of IP addresses separated by commas. API clients with IP
 # addresses in this list are able to connect to the API without Gateway
 # generating the 'Incoming connection' popup.
 #
 # Note that 127.0.0.1 is always permitted to connect, so do not include it
 # in this setting.
 TrustedTwsApiClientIPs=
 # Reset Order ID Sequence
 # -----------------------
 #
 # The setting resets the order id sequence for orders submitted via the API, so
 # that the next invocation of the `NextValidId` API callback will return the
 # value 1. The reset occurs when TWS starts.
 #
 # Note that order ids are reset for all API clients, except those that have
 # outstanding (ie incomplete) orders: their order id sequence carries on as
 # before.
 #
 # Valid values are 'yes', 'true', 'false' and 'no'. The default is 'no'.
 ResetOrderIdsAtStart=
 # This setting specifies IBC's action when TWS displays the dialog asking for
 # confirmation of a request to reset the API order id sequence.
 #
 # Note that the Gateway never displays this dialog, so this setting is ignored
 # for a Gateway session.
 #
 # Valid values consist of two strings separated by a solidus '/'. The first
 # value specifies the action to take when the order id reset request resulted
 # from setting ResetOrderIdsAtStart=yes. The second specifies the action to
 # take when the order id reset request is a result of the user clicking the
 # 'Reset API order ID sequence' button in the API configuration. Each value
 # must be one of the following:
 #
 #    'confirm' 
 #        order ids will be reset
 #
 #    'reject' 
 #        order ids will not be reset
 #
 #    'ignore' 
 #        IBC will ignore the dialog. The user must take action.
 #
 #    The default setting is ignore/ignore
 # Examples:
 #
 #    'confirm/reject' - confirm order id reset only if ResetOrderIdsAtStart=yes
 #                       and reject any user-initiated requests
 #
 #    'ignore/confirm' - user must decide what to do if ResetOrderIdsAtStart=yes
 #                       and confirm user-initiated requests
 #
 #    'reject/ignore'  - reject order id reset if  ResetOrderIdsAtStart=yes but
 #                       allow user to handle user-initiated requests 
 ConfirmOrderIdReset=
 # =============================================================================
-# 4.   TWS Auto-Logoff and Auto-Restart
+# 4.   TWS Auto-Closedown
 # =============================================================================
 #
-# TWS and Gateway insist on being restarted every day. Two alternative
+# IMPORTANT NOTE: Starting with TWS 974, this setting no longer
-# automatic options are offered: 
+# works properly, because IB have changed the way TWS handles its
 # autologoff mechanism. 
 #
-#    - Auto-Logoff: at a specified time, TWS shuts down tidily, without
+# You should now configure the TWS autologoff time to something
-#      restarting.
+# convenient for you, and restart IBC each day.
 #
-#    - Auto-Restart: at a specified time, TWS shuts down and then restarts
+# Alternatively, discontinue use of IBC and use the auto-relogin
-#      without the user having to re-autheticate.
+# mechanism within TWS 974 and later versions (note that the 
-#
+# auto-relogin mechanism provided by IB is not available if you
-# The normal way to configure the time at which this happens is via the Lock
+# use IBC).
 # and Exit section of the Configuration dialog. Once this time has been
 # configured in this way, the setting persists until the user changes it again.
 #
 # However, there are situations where there is no user available to do this
 # configuration, or where there is no persistent storage (for example some
 # Docker images). In such cases, the auto-restart or auto-logoff time can be
 # set whenever IBC starts with the settings below.
 #
 # The value, if specified, must be a time in HH:MM AM/PM format, for example
 # 08:00 AM or 10:00 PM. Note that there must be a single space between the
 # two parts of this value; also that midnight is "12:00 AM" and midday is
 # "12:00 PM".
 #
 # If no value is specified for either setting, the currently configured
 # settings will apply. If a value is supplied for one setting, the other
 # setting is cleared. If values are supplied for both settings, only the
 # auto-restart time is set, and the auto-logoff time is cleared.
 #
 # Note that for a normal TWS/Gateway installation with persistent storage
 # (for example on a desktop computer) the value will be persisted as if the
 # user had set it via the configuration dialog.
 #
 # If you choose to auto-restart, you should take note of the considerations
 # described at the link below. Note that where this information mentions
 # 'manual authentication', restarting IBC will do the job (IBKR does not
 # recognise the existence of IBC in its docuemntation).
 #
 #  https://www.interactivebrokers.com/en/software/tws/twsguide.htm#usersguidebook/configuretws/auto_restart_info.htm
 #
 # If you use the "RESTART" command via the IBC command server, and IBC is
 # running any version of the Gateway (or a version of TWS earlier than 1018),
 # note that this will set the Auto-Restart time in Gateway/TWS's configuration
 # dialog to the time at which the restart actually happens (which may be up to
 # a minute after the RESTART command is issued). To prevent future auto-
 # restarts at this time, you must make sure you have set AutoLogoffTime or
 # AutoRestartTime to your desired value before running IBC. NB: this does not
 # apply to TWS from version 1018 onwards.
-AutoLogoffTime=
+# Set to yes or no (lower case).
 #
 #   yes   means allow TWS to shut down automatically at its
 # 	  specified shutdown time, which is set via the TWS
 #	  configuration menu.
 #
 #   no    means TWS never shuts down automatically.
 #
 # NB: IB recommends that you do not keep TWS running
 # continuously. If you set this setting to 'no', you may
 # experience incorrect TWS operation.
 #
 # NB: the default for this setting is 'no'. Since this will
 # only work properly with TWS versions earlier than 974, you
 # should explicitly set this to 'yes' for version 974 and later.
 IbAutoClosedown=yes
 AutoRestartTime=
 # =============================================================================
 # 5.   TWS Tidy Closedown Time
 # =============================================================================
 #
-# Specifies a time at which TWS will close down tidily, with no restart.
+# NB: starting with TWS 974 this is no longer a useful option
 # because both TWS and Gateway now have the same auto-logoff
 # mechanism, and IBC can no longer avoid this.
 #
-# There is little reason to use this setting. It is similar to AutoLogoffTime,
+# Note that giving this setting a value does not change TWS's
-# but can include a day-of-the-week, whereas AutoLogoffTime and AutoRestartTime
+# auto-logoff in any way: any setting will be additional to the
-# apply every day. So for example you could use ClosedownAt in conjunction with
+# TWS auto-logoff.
 # AutoRestartTime to shut down TWS on Friday evenings after the markets
 # close, without it running on Saturday as well.
 #
 # To tell IBC to tidily close TWS at a specified time every
 # day, set this value to <hh:mm>, for example:
@ -679,7 +487,7 @@ AcceptIncomingConnectionAction=reject
 #   no    means the dialog remains on display and must be
 #         handled by the user.
-AllowBlindTrading=no
+AllowBlindTrading=yes
 # Save Settings on a Schedule
@ -722,26 +530,6 @@ AllowBlindTrading=no
 SaveTwsSettingsAt=
 # Confirm Crypto Currency Orders Automatically
 # --------------------------------------------
 #
 # When you place an order for a cryptocurrency contract, a dialog is displayed
 # asking you to confirm that you want to place the order, and notifying you
 # that you are placing an order to trade cryptocurrency with Paxos, a New York
 # limited trust company, and not at Interactive Brokers.
 #
 #   transmit    means that the order will be placed automatically, and the
 #               dialog will then be closed
 #
 #   cancel      means that the order will not be placed, and the dialog will
 #               then be closed
 #
 #   manual      means that IBC will take no action and the user must deal
 #               with the dialog
 ConfirmCryptoCurrencyOrders=transmit
 # =============================================================================
 # 7.   Settings Specific to Indian Versions of TWS
@ -778,17 +566,13 @@ DismissNSEComplianceNotice=yes
 #
 # The port number that IBC listens on for commands
 # such as "STOP". DO NOT set this to the port number
-# used for TWS API connections.
+# used for TWS API connections. There is no good reason
-#
+# to change this setting unless the port is used by
-# The convention is to use 7462 for this port,
+# some other application (typically another instance of
-# but it must be set to a different value from any other
+# IBC). The default value is 0, which tells IBC not to
-# IBC instance that might run at the same time.
+# start the command server
 #
 # The default value is 0, which tells IBC not to start
 # the command server
 #CommandServerPort=7462
 CommandServerPort=0
 # Permitted Command Sources
@ -799,19 +583,19 @@ CommandServerPort=0
 # IBC.  Commands can always be sent from the
 # same host as IBC is running on.
-ControlFrom=
+ControlFrom=127.0.0.1
 # Address for Receiving Commands
 # ------------------------------
 #
 # Specifies the IP address on which the Command Server
-# is to listen. For a multi-homed host, this can be used
+# is so listen. For a multi-homed host, this can be used
 # to specify that connection requests are only to be
 # accepted on the specified address. The default is to
 # accept connection requests on all local addresses.
-BindAddress=
+BindAddress=127.0.0.1
 # Command Prompt
@ -837,7 +621,7 @@ CommandPrompt=
 # information is sent. The default is that such information
 # is not sent.
-SuppressInfoMessages=yes
+SuppressInfoMessages=no
@ -867,10 +651,10 @@ SuppressInfoMessages=yes
 # The LogStructureScope setting indicates which windows are
 # eligible for structure logging:
 #
-#    - (default value) if set to 'known', only windows that
+#    - if set to 'known', only windows that IBC recognizes
-#      IBC recognizes are eligible - these are windows that
+#      are eligible - these are windows that IBC has some
-#      IBC has some interest in monitoring, usually to take
+#      interest in monitoring, usually to take some action
-#      some action on the user's behalf;
+#      on the user's behalf;
 #
 #    - if set to 'unknown', only windows that IBC does not
 #      recognize are eligible. Most windows displayed by
@ -883,8 +667,9 @@ SuppressInfoMessages=yes
 #    - if set to 'all', then every window displayed by TWS
 #      is eligible.
 #
 # The default value is 'known'.
-LogStructureScope=known
+LogStructureScope=all
 # When to Log Window Structure
@ -897,15 +682,13 @@ LogStructureScope=known
 #       structure of an eligible window the first time it
 #       is encountered;
 #
 #     - if set to 'openclose', the structure is logged every
 #       time an eligible window is opened or closed;
 #
 #    - if set to 'activate', the structure is logged every
 #      time an eligible window is made active;
 #
-#    - (default value) if set to 'never' or 'no' or 'false',
+#    - if set to 'never' or 'no' or 'false', structure
-#      structure information is never logged.
+#      information is never logged.
 #
 # The default value is 'never'.
 LogStructureWhen=never
@ -925,3 +708,4 @@ LogStructureWhen=never
 #LogComponents=
--- a/docs/tina_readme.rst
+++ b/docs/tina_readme.rst
@ -1,91 +0,0 @@
 ### NOTE this is likely out of date given it was written some
 (years) time ago by a user that has since not really partaken in
 contributing since.
 install for tinas
 *****************
 for windows peeps you can start by installing all the prerequisite software:
 - install git with all default settings - https://git-scm.com/download/win
 - install anaconda all default settings - https://www.anaconda.com/products/individual
 - install microsoft build tools (check the box for Desktop development for C++, you might be able to uncheck some optional downloads)  - https://visualstudio.microsoft.com/visual-cpp-build-tools/
 - install visual studio code default settings - https://code.visualstudio.com/download
 then, `crack a conda shell`_ and run the following commands::
    mkdir code # create code directory
    cd code # change directory to code
    git clone https://github.com/pikers/piker.git # downloads piker installation package from github
    cd piker # change directory to piker
    conda create -n pikonda # creates conda environment named pikonda
    conda activate pikonda # activates pikonda
    conda install -c conda-forge python-levenshtein # in case it is not already installed
    conda install pip # may already be installed
    pip # will show if pip is installed
    pip install -e . -r requirements.txt # install piker in editable mode
 test Piker to see if it is working::
    piker -b binance chart btcusdt.binance # formatting for loading a chart
    piker -b kraken -b binance chart xbtusdt.kraken
    piker -b kraken -b binance -b ib chart qqq.nasdaq.ib
    piker -b ib chart tsla.nasdaq.ib
 potential error::
    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Roaming\\piker\\brokers.toml'
 solution:
 - navigate to file directory above (may be different on your machine, location should be listed in the error code)
 - copy and paste file from 'C:\\Users\\user\\code\\data/brokers.toml' or create a blank file using notepad at the location above
 Visual Studio Code setup:
 - now that piker is installed we can set up vscode as the default terminal for running piker and editing the code
 - open Visual Studio Code
 - file --> Add Folder to Workspace --> C:\Users\user\code\piker (adds piker directory where all piker files are located)
 - file --> Save Workspace As --> save it wherever you want and call it whatever you want, this is going to be your default workspace for running and editing piker code
 - ctrl + shift + p --> start typing Python: Select Interpetter --> when the option comes up select it --> Select at the workspace level --> select the one that shows ('pikonda')
 - change the default terminal to cmd.exe instead of powershell (default)
 - now when you create a new terminal VScode should automatically activate you conda env so that piker can be run as the first command after a new terminal is created
 also, try out fancyzones as part of powertoyz for a decent tiling windows manager to manage all the cool new software you are going to be running.
 .. _conda installed: https://
 .. _C++ build toolz: https://
 .. _crack a conda shell: https://
 .. _vscode: https://
 .. link to the tina guide
 .. _setup a coolio tiled wm console: https://
 provider support
 ****************
 for live data feeds the in-progress set of supported brokers is:
 - IB_ via ``ib_insync``, also see our `container docs`_
 - binance_ and kraken_ for crypto over their public websocket API
 - questrade_ (ish) which comes with effectively free L1
 coming soon...
 - webull_ via the reverse engineered public API
 - yahoo via yliveticker_
 if you want your broker supported and they have an API let us know.
 .. _IB: https://interactivebrokers.github.io/tws-api/index.html
 .. _container docs: https://github.com/pikers/piker/tree/master/dockering/ib
 .. _questrade: https://www.questrade.com/api/documentation
 .. _kraken: https://www.kraken.com/features/api#public-market-data
 .. _binance: https://github.com/pikers/piker/pull/182
 .. _webull: https://github.com/tedchou12/webull
 .. _yliveticker: https://github.com/yahoofinancelive/yliveticker
 .. _coinbase: https://docs.pro.coinbase.com/#websocket-feed
--- a/examples/basic_order_bot.py
+++ b/examples/basic_order_bot.py
@ -1,264 +0,0 @@
 # from pprint import pformat
 from functools import partial
 from decimal import Decimal
 from typing import Callable
 import tractor
 import trio
 from uuid import uuid4
 from piker.service import maybe_open_pikerd
 from piker.accounting import dec_digits
 from piker.clearing import (
    open_ems,
    OrderClient,
 )
 # TODO: we should probably expose these top level in this subsys?
 from piker.clearing._messages import (
    Order,
    Status,
    BrokerdPosition,
 )
 from piker.data import (
    iterticks,
    Flume,
    open_feed,
    Feed,
    # ShmArray,
 )
 # TODO: handle other statuses:
 # - fills, errors, and position tracking
 async def wait_for_order_status(
    trades_stream: tractor.MsgStream,
    oid: str,
    expect_status: str,
 ) -> tuple[
    list[Status],
    list[BrokerdPosition],
 ]:
    '''
    Wait for a specific order status for a given dialog, return msg flow
    up to that msg and any position update msgs in a tuple.
    '''
    # Wait for position message before moving on to verify flow(s)
    # for the multi-order position entry/exit.
    status_msgs: list[Status] = []
    pp_msgs: list[BrokerdPosition] = []
    async for msg in trades_stream:
        match msg:
            case {'name': 'position'}:
                ppmsg = BrokerdPosition(**msg)
                pp_msgs.append(ppmsg)
            case {
                'name': 'status',
            }:
                msg = Status(**msg)
                status_msgs.append(msg)
                # if we get the status we expect then return all
                # collected msgs from the brokerd dialog up to the
                # exected msg B)
                if (
                     msg.resp == expect_status
                     and msg.oid == oid
                ):
                    return status_msgs, pp_msgs
 async def bot_main():
    '''
    Boot the piker runtime, open an ems connection, submit
    and process orders statuses in real-time.
    '''
    ll: str = 'info'
    # open an order ctl client, live data feed, trio nursery for
    # spawning an order trailer task
    client: OrderClient
    trades_stream: tractor.MsgStream
    feed: Feed
    accounts: list[str]
    fqme: str = 'btcusdt.usdtm.perp.binance'
    async with (
        # TODO: do this implicitly inside `open_ems()` ep below?
        # init and sync actor-service runtime
        maybe_open_pikerd(
            loglevel=ll,
            debug_mode=True,
        ),
        open_ems(
            fqme,
            mode='paper',  # {'live', 'paper'}
            # mode='live',  # for real-brokerd submissions
            loglevel=ll,
        ) as (
            client,  # OrderClient
            trades_stream,  # tractor.MsgStream startup_pps,
            _,  # positions
            accounts,
            _,  # dialogs
        ),
        open_feed(
            fqmes=[fqme],
            loglevel=ll,
            # TODO: if you want to throttle via downsampling
            # how many tick updates your feed received on
            # quote streams B)
            # tick_throttle=10,
        ) as feed,
        tractor.trionics.collapse_eg(),
        trio.open_nursery() as tn,
    ):
        assert accounts
        print(f'Loaded binance accounts: {accounts}')
        flume: Flume = feed.flumes[fqme]
        min_tick = Decimal(flume.mkt.price_tick)
        min_tick_digits: int = dec_digits(min_tick)
        price_round: Callable = partial(
            round,
            ndigits=min_tick_digits,
        )
        quote_stream: trio.abc.ReceiveChannel = feed.streams['binance']
        # always keep live limit 0.003% below last
        # clearing price
        clear_margin: float = 0.9997
        async def trailer(
            order: Order,
        ):
            # ref shm OHLCV array history, if you want
            # s_shm: ShmArray = flume.rt_shm
            # m_shm: ShmArray = flume.hist_shm
            # NOTE: if you wanted to frame ticks by type like the
            # the quote throttler does.. and this is probably
            # faster in terms of getting the latest tick type
            # embedded value of interest?
            # from piker.data._sampling import frame_ticks
            async for quotes in quote_stream:
                for fqme, quote in quotes.items():
                    # print(
                    #     f'{quote["symbol"]} -> {quote["ticks"]}\n'
                    #     f'last 1s OHLC:\n{s_shm.array[-1]}\n'
                    #     f'last 1m OHLC:\n{m_shm.array[-1]}\n'
                    # )
                    for tick in iterticks(
                        quote,
                        reverse=True,
                        # types=('trade', 'dark_trade'), # defaults
                    ):
                        await client.update(
                            uuid=order.oid,
                            price=price_round(
                                clear_margin
                                *
                                tick['price']
                            ),
                        )
                        msgs, pps = await wait_for_order_status(
                            trades_stream,
                            order.oid,
                            'open'
                        )
                        # if multiple clears per quote just
                        # skip to the next quote?
                        break
        # get first live quote to be sure we submit the initial
        # live buy limit low enough that it doesn't clear due to
        # a stale initial price from the data feed layer!
        first_ask_price: float | None = None
        async for quotes in quote_stream:
            for fqme, quote in quotes.items():
                # print(quote['symbol'])
                for tick in iterticks(quote, types=('ask')):
                    first_ask_price: float = tick['price']
                    break
            if first_ask_price:
                break
        # setup order dialog via first msg
        price: float = price_round(
                clear_margin
                *
                first_ask_price,
        )
        # compute a 1k USD sized pos
        size: float = round(1e3/price, ndigits=3)
        order = Order(
            # docs on how this all works, bc even i'm not entirely
            # clear XD. also we probably want to figure  out how to
            # offer both the paper engine running and the brokerd
            # order ctl tasks with the ems choosing which stream to
            # route msgs on given the account value!
            account='paper',  # use built-in paper clearing engine and .accounting
            # account='binance.usdtm',  # for live binance futes
            oid=str(uuid4()),
            exec_mode='live',  # {'dark', 'live', 'alert'}
            action='buy',  # TODO: remove this from our schema?
            size=size,
            symbol=fqme,
            price=price,
            brokers=['binance'],
        )
        await client.send(order)
        msgs, pps = await wait_for_order_status(
            trades_stream,
            order.oid,
            'open',
        )
        assert not pps
        assert msgs[-1].oid == order.oid
        # start "trailer task" which tracks rt quote stream
        tn.start_soon(trailer, order)
        try:
            # wait for ctl-c from user..
            await trio.sleep_forever()
        except KeyboardInterrupt:
            # cancel the open order
            await client.cancel(order.oid)
            msgs, pps = await wait_for_order_status(
                trades_stream,
                order.oid,
                'canceled'
            )
            raise
 if __name__ == '__main__':
    trio.run(bot_main)
--- a/flake.lock
+++ b/flake.lock
@ -1,27 +0,0 @@
 {
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1765779637,
        "narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
        "owner": "nixos",
        "repo": "nixpkgs",
        "rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
        "type": "github"
      },
      "original": {
        "owner": "nixos",
        "ref": "nixos-unstable",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
 }
--- a/flake.nix
+++ b/flake.nix
@ -1,103 +0,0 @@
 # An "impure" template thx to `pyproject.nix`,
 # https://pyproject-nix.github.io/pyproject.nix/templates.html#impure
 # https://github.com/pyproject-nix/pyproject.nix/blob/master/templates/impure/flake.nix
 {
  description = "An impure `piker` overlay using `uv` with Nix(OS)";
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };
  outputs =
    { nixpkgs, ... }:
    let
      inherit (nixpkgs) lib;
      forAllSystems = lib.genAttrs lib.systems.flakeExposed;
    in
    {
      devShells = forAllSystems (
        system:
        let
          pkgs = nixpkgs.legacyPackages.${system};
          # do store-path extractions
          qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
          # ?TODO? can remove below since manual linking not needed?
          # qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
          # XXX NOTE XXX, for now we overlay specific pkgs via
          # a major-version-pinned-`cpython`
          cpython = "python313";
          pypkgs = pkgs."${cpython}Packages";
        in
        {
          default = pkgs.mkShell {
            packages = with pkgs; [
              # XXX, ensure sh completions active!
              bashInteractive
              bash-completion
              # dev utils
              ruff
              pypkgs.ruff
              qt6.qtwayland
              qt6.qtbase
              uv
              python313  # ?TODO^ how to set from `cpython` above?
              pypkgs.pyqt6
              pypkgs.pyqt6-sip
              pypkgs.qtpy
              pypkgs.qdarkstyle
              pypkgs.rapidfuzz
            ];
            shellHook = ''
              # unmask to debug **this** dev-shell-hook
              # set -e
              # set qt-base/plugin path(s)
              QTBASE_PATH="${qt6baseStorePath}/lib"
              QT_PLUGIN_PATH="${qt6baseStorePath}/lib/qt-6/plugins"
              QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
              # link in Qt cc lib paths from <nixpkgs>
              LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
              LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
              LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
              # link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
              LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
              export LD_LIBRARY_PATH
              # RUNTIME-SETTINGS
              #
              # ------ Qt ------
              # XXX, unmask to debug qt .so linking/loading deats
              # export QT_DEBUG_PLUGINS=1
              #
              # ALSO, for *modern linux* DEs,
              # - maybe set wayland-mode (TODO, parametrtize this!)
              #  * a chosen wayland-mode shell-integration
              export QT_QPA_PLATFORM="wayland"
              export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
              # ------ uv ------
              # - always use the ./py313/ venv-subdir
              export UV_PROJECT_ENVIRONMENT="py313"
              # sync project-env with all extras
              uv sync --dev --all-extras --no-group lint
              # ------ TIPS ------
              # NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
              # run the `nix develop` cmd with,
              # >> nix develop -c uv run xonsh
            '';
          };
        }
      );
    };
 }
--- a/notes_to_self/ideas.rst
+++ b/notes_to_self/ideas.rst
--- a/piker/init.py
+++ b/piker/init.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers.
-# Copyright 2020-eternity Tyler Goodlet (in stewardship for pikers)
+# Copyright 2020-eternity Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -14,11 +14,11 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
 piker: trading gear for hackers.
-'''
+"""
-from .service import open_piker_runtime
+from ._daemon import open_piker_runtime
 from .data.feed import open_feed
 __all__ = [
--- a/piker/_cacheables.py
+++ b/piker/_cacheables.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -14,71 +14,37 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
 Cacheing apis and toolz.
-'''
+"""
 from collections import OrderedDict
-from typing import (
+from contextlib import (
-    Awaitable,
+    asynccontextmanager,
    Callable,
    ParamSpec,
    TypeVar,
 )
 from tractor.trionics import maybe_open_context
 from .brokers import get_brokermod
 from .log import get_logger
 log = get_logger(__name__)
 T = TypeVar("T")
 P = ParamSpec("P")
-
+def async_lifo_cache(maxsize=128):
-# TODO: move this to `tractor.trionics`..
+    """Async ``cache`` with a LIFO policy.
 # - egs. to replicate for tests: https://github.com/aio-libs/async-lru#usage
 # - their suite as well:
 #   https://github.com/aio-libs/async-lru/tree/master/tests
 # - asked trio_util about it too:
 #   https://github.com/groove-x/trio-util/issues/21
 def async_lifo_cache(
    maxsize=128,
    # NOTE: typing style was learned from:
    # https://stackoverflow.com/a/71132186
 ) -> Callable[
    Callable[P, Awaitable[T]],
    Callable[
        Callable[P, Awaitable[T]],
        Callable[P, Awaitable[T]],
    ],
 ]:
    '''
    Async ``cache`` with a LIFO policy.
    Implemented my own since no one else seems to have
    a standard. I'll wait for the smarter people to come
    up with one, but until then...
-
+    """
    NOTE: when decorating, due to this simple/naive implementation, you
    MUST call the decorator like,
    .. code:: python
        @async_lifo_cache()
        async def cache_target():
    '''
    cache = OrderedDict()
-    def decorator(
+    def decorator(fn):
        fn: Callable[P, Awaitable[T]],
    ) -> Callable[P, Awaitable[T]]:
-        async def decorated(
+        async def wrapper(*args):
            *args: P.args,
            **kwargs: P.kwargs,
        ) -> T:
            key = args
            try:
                return cache[key]
@ -87,13 +53,27 @@ def async_lifo_cache(
                    # discard last added new entry
                    cache.popitem()
-                # call underlying
+                # do it
-                cache[key] = await fn(
+                cache[key] = await fn(*args)
                    *args,
                    **kwargs,
                )
                return cache[key]
-        return decorated
+        return wrapper
    return decorator
@asynccontextmanager
 async def open_cached_client(
    brokername: str,
 ) -> 'Client':  # noqa
    '''
    Get a cached broker client from the current actor's local vars.
    If one has not been setup do it and cache it.
    '''
    brokermod = get_brokermod(brokername)
    async with maybe_open_context(
        acm_func=brokermod.get_client,
    ) as (cache_hit, client):
        yield client
--- a/piker/_daemon.py
+++ b/piker/_daemon.py
@ -0,0 +1,704 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 Structured, daemon tree service management.
 """
 from __future__ import annotations
 import os
 from typing import (
    Optional,
    Callable,
    Any,
    ClassVar,
 )
 from contextlib import (
    asynccontextmanager as acm,
 )
 from collections import defaultdict
 import tractor
 import trio
 from trio_typing import TaskStatus
 from .log import (
    get_logger,
    get_console_log,
 )
 from .brokers import get_brokermod
 log = get_logger(__name__)
 _root_dname = 'pikerd'
 _default_registry_host: str = '127.0.0.1'
 _default_registry_port: int = 6116
 _default_reg_addr: tuple[str, int] = (
    _default_registry_host,
    _default_registry_port,
 )
 # NOTE: this value is set as an actor-global once the first endpoint
 # who is capable, spawns a `pikerd` service tree.
 _registry: Registry | None = None
 class Registry:
    addr: None | tuple[str, int] = None
    # TODO: table of uids to sockaddrs
    peers: dict[
        tuple[str, str],
        tuple[str, int],
    ] = {}
 _tractor_kwargs: dict[str, Any] = {}
@acm
 async def open_registry(
    addr: None | tuple[str, int] = None,
    ensure_exists: bool = True,
 ) -> tuple[str, int]:
    global _tractor_kwargs
    actor = tractor.current_actor()
    uid = actor.uid
    if (
        Registry.addr is not None
        and addr
    ):
        raise RuntimeError(
            f'`{uid}` registry addr already bound @ {_registry.sockaddr}'
        )
    was_set: bool = False
    if (
        not tractor.is_root_process()
        and Registry.addr is None
    ):
        Registry.addr = actor._arb_addr
    if (
        ensure_exists
        and Registry.addr is None
    ):
        raise RuntimeError(
            f"`{uid}` registry should already exist bug doesn't?"
        )
    if (
        Registry.addr is None
    ):
        was_set = True
        Registry.addr = addr or _default_reg_addr
    _tractor_kwargs['arbiter_addr'] = Registry.addr
    try:
        yield Registry.addr
    finally:
        # XXX: always clear the global addr if we set it so that the
        # next (set of) calls will apply whatever new one is passed
        # in.
        if was_set:
            Registry.addr = None
 def get_tractor_runtime_kwargs() -> dict[str, Any]:
    '''
    Deliver ``tractor`` related runtime variables in a `dict`.
    '''
    return _tractor_kwargs
 _root_modules = [
    __name__,
    'piker.clearing._ems',
    'piker.clearing._client',
    'piker.data._sampling',
 ]
 # TODO: factor this into a ``tractor.highlevel`` extension
 # pack for the library.
 class Services:
    actor_n: tractor._supervise.ActorNursery
    service_n: trio.Nursery
    debug_mode: bool  # tractor sub-actor debug mode flag
    service_tasks: dict[
        str,
        tuple[
            trio.CancelScope,
            tractor.Portal,
            trio.Event,
        ]
    ] = {}
    locks = defaultdict(trio.Lock)
    @classmethod
    async def start_service_task(
        self,
        name: str,
        portal: tractor.Portal,
        target: Callable,
        **kwargs,
    ) -> (trio.CancelScope, tractor.Context):
        '''
        Open a context in a service sub-actor, add to a stack
        that gets unwound at ``pikerd`` teardown.
        This allows for allocating long-running sub-services in our main
        daemon and explicitly controlling their lifetimes.
        '''
        async def open_context_in_task(
            task_status: TaskStatus[
                tuple[
                    trio.CancelScope,
                    trio.Event,
                    Any,
                ]
            ] = trio.TASK_STATUS_IGNORED,
        ) -> Any:
            with trio.CancelScope() as cs:
                async with portal.open_context(
                    target,
                    **kwargs,
                ) as (ctx, first):
                    # unblock once the remote context has started
                    complete = trio.Event()
                    task_status.started((cs, complete, first))
                    log.info(
                        f'`pikerd` service {name} started with value {first}'
                    )
                    try:
                        # wait on any context's return value
                        # and any final portal result from the
                        # sub-actor.
                        ctx_res = await ctx.result()
                        # NOTE: blocks indefinitely until cancelled
                        # either by error from the target context
                        # function or by being cancelled here by the
                        # surrounding cancel scope.
                        return (await portal.result(), ctx_res)
                    finally:
                        await portal.cancel_actor()
                        complete.set()
                        self.service_tasks.pop(name)
        cs, complete, first = await self.service_n.start(open_context_in_task)
        # store the cancel scope and portal for later cancellation or
        # retstart if needed.
        self.service_tasks[name] = (cs, portal, complete)
        return cs, first
    @classmethod
    async def cancel_service(
        self,
        name: str,
    ) -> Any:
        '''
        Cancel the service task and actor for the given ``name``.
        '''
        log.info(f'Cancelling `pikerd` service {name}')
        cs, portal, complete = self.service_tasks[name]
        cs.cancel()
        await complete.wait()
        assert name not in self.service_tasks, \
            f'Serice task for {name} not terminated?'
@acm
 async def open_piker_runtime(
    name: str,
    enable_modules: list[str] = [],
    loglevel: Optional[str] = None,
    # XXX NOTE XXX: you should pretty much never want debug mode
    # for data daemons when running in production.
    debug_mode: bool = False,
    registry_addr: None | tuple[str, int] = None,
    # TODO: once we have `rsyscall` support we will read a config
    # and spawn the service tree distributed per that.
    start_method: str = 'trio',
    tractor_kwargs: dict = {},
 ) -> tuple[
    tractor.Actor,
    tuple[str, int],
 ]:
    '''
    Start a piker actor who's runtime will automatically sync with
    existing piker actors on the local link based on configuration.
    Can be called from a subactor or any program that needs to start
    a root actor.
    '''
    try:
        # check for existing runtime
        actor = tractor.current_actor().uid
    except tractor._exceptions.NoRuntime:
        registry_addr = registry_addr or _default_reg_addr
        async with (
            tractor.open_root_actor(
                # passed through to ``open_root_actor``
                arbiter_addr=registry_addr,
                name=name,
                loglevel=loglevel,
                debug_mode=debug_mode,
                start_method=start_method,
                # TODO: eventually we should be able to avoid
                # having the root have more then permissions to
                # spawn other specialized daemons I think?
                enable_modules=enable_modules,
                **tractor_kwargs,
            ) as _,
            open_registry(registry_addr, ensure_exists=False) as addr,
        ):
            yield (
                tractor.current_actor(),
                addr,
            )
    else:
        async with open_registry(registry_addr) as addr:
            yield (
                actor,
                addr,
            )
@acm
 async def open_pikerd(
    loglevel: str | None = None,
    # XXX: you should pretty much never want debug mode
    # for data daemons when running in production.
    debug_mode: bool = False,
    registry_addr: None | tuple[str, int] = None,
 ) -> Services:
    '''
    Start a root piker daemon who's lifetime extends indefinitely until
    cancelled.
    A root actor nursery is created which can be used to create and keep
    alive underling services (see below).
    '''
    async with (
        open_piker_runtime(
            name=_root_dname,
            # TODO: eventually we should be able to avoid
            # having the root have more then permissions to
            # spawn other specialized daemons I think?
            enable_modules=_root_modules,
            loglevel=loglevel,
            debug_mode=debug_mode,
            registry_addr=registry_addr,
        ) as (root_actor, reg_addr),
        tractor.open_nursery() as actor_nursery,
        trio.open_nursery() as service_nursery,
    ):
        assert root_actor.accept_addr == reg_addr
        # assign globally for future daemon/task creation
        Services.actor_n = actor_nursery
        Services.service_n = service_nursery
        Services.debug_mode = debug_mode
        try:
            yield Services
        finally:
            # TODO: is this more clever/efficient?
            # if 'samplerd' in Services.service_tasks:
            #     await Services.cancel_service('samplerd')
            service_nursery.cancel_scope.cancel()
@acm
 async def maybe_open_runtime(
    loglevel: Optional[str] = None,
    **kwargs,
 ) -> None:
    '''
    Start the ``tractor`` runtime (a root actor) if none exists.
    '''
    name = kwargs.pop('name')
    if not tractor.current_actor(err_on_no_runtime=False):
        async with open_piker_runtime(
            name,
            loglevel=loglevel,
            **kwargs,
        ) as (_, addr):
            yield addr,
    else:
        async with open_registry() as addr:
            yield addr
@acm
 async def maybe_open_pikerd(
    loglevel: Optional[str] = None,
    registry_addr: None | tuple = None,
    **kwargs,
 ) -> tractor._portal.Portal | ClassVar[Services]:
    '''
    If no ``pikerd`` daemon-root-actor can be found start it and
    yield up (we should probably figure out returning a portal to self
    though).
    '''
    if loglevel:
        get_console_log(loglevel)
    # subtle, we must have the runtime up here or portal lookup will fail
    query_name = kwargs.pop('name', f'piker_query_{os.getpid()}')
    # TODO: if we need to make the query part faster we could not init
    # an actor runtime and instead just hit the socket?
    # from tractor._ipc import _connect_chan, Channel
    # async with _connect_chan(host, port) as chan:
    #     async with open_portal(chan) as arb_portal:
    #         yield arb_portal
    async with (
        open_piker_runtime(
            name=query_name,
            registry_addr=registry_addr,
            loglevel=loglevel,
            **kwargs,
        ) as _,
        tractor.find_actor(
            _root_dname,
            arbiter_sockaddr=registry_addr,
        ) as portal
    ):
        # connect to any existing daemon presuming
        # its registry socket was selected.
        if (
            portal is not None
        ):
            yield portal
            return
    # presume pikerd role since no daemon could be found at
    # configured address
    async with open_pikerd(
        loglevel=loglevel,
        debug_mode=kwargs.get('debug_mode', False),
        registry_addr=registry_addr,
    ) as service_manager:
        # in the case where we're starting up the
        # tractor-piker runtime stack in **this** process
        # we return no portal to self.
        assert service_manager
        yield service_manager
 # `brokerd` enabled modules
 # NOTE: keeping this list as small as possible is part of our caps-sec
 # model and should be treated with utmost care!
 _data_mods = [
    'piker.brokers.core',
    'piker.brokers.data',
    'piker.data',
    'piker.data.feed',
    'piker.data._sampling'
 ]
@acm
 async def find_service(
    service_name: str,
 ) -> tractor.Portal | None:
    async with open_registry() as reg_addr:
        log.info(f'Scanning for service `{service_name}`')
        # attach to existing daemon by name if possible
        async with tractor.find_actor(
            service_name,
            arbiter_sockaddr=reg_addr,
        ) as maybe_portal:
            yield maybe_portal
 async def check_for_service(
    service_name: str,
 ) -> None | tuple[str, int]:
    '''
    Service daemon "liveness" predicate.
    '''
    async with open_registry(ensure_exists=False) as reg_addr:
        async with tractor.query_actor(
            service_name,
            arbiter_sockaddr=reg_addr,
        ) as sockaddr:
            return sockaddr
@acm
 async def maybe_spawn_daemon(
    service_name: str,
    service_task_target: Callable,
    spawn_args: dict[str, Any],
    loglevel: Optional[str] = None,
    singleton: bool = False,
    **kwargs,
 ) -> tractor.Portal:
    '''
    If no ``service_name`` daemon-actor can be found,
    spawn one in a local subactor and return a portal to it.
    If this function is called from a non-pikerd actor, the
    spawned service will persist as long as pikerd does or
    it is requested to be cancelled.
    This can be seen as a service starting api for remote-actor
    clients.
    '''
    if loglevel:
        get_console_log(loglevel)
    # serialize access to this section to avoid
    # 2 or more tasks racing to create a daemon
    lock = Services.locks[service_name]
    await lock.acquire()
    async with find_service(service_name) as portal:
        if portal is not None:
            lock.release()
            yield portal
            return
    log.warning(f"Couldn't find any existing {service_name}")
    # TODO: really shouldn't the actor spawning be part of the service
    # starting method `Services.start_service()` ?
    # ask root ``pikerd`` daemon to spawn the daemon we need if
    # pikerd is not live we now become the root of the
    # process tree
    async with maybe_open_pikerd(
        loglevel=loglevel,
        **kwargs,
    ) as pikerd_portal:
        # we are the root and thus are `pikerd`
        # so spawn the target service directly by calling
        # the provided target routine.
        # XXX: this assumes that the target is well formed and will
        # do the right things to setup both a sub-actor **and** call
        # the ``_Services`` api from above to start the top level
        # service task for that actor.
        started: bool
        if pikerd_portal is None:
            started = await service_task_target(**spawn_args)
        else:
            # tell the remote `pikerd` to start the target,
            # the target can't return a non-serializable value
            # since it is expected that service startingn is
            # non-blocking and the target task will persist running
            # on `pikerd` after the client requesting it's start
            # disconnects.
            started = await pikerd_portal.run(
                service_task_target,
                **spawn_args,
            )
        if started:
            log.info(f'Service {service_name} started!')
        async with tractor.wait_for_actor(service_name) as portal:
            lock.release()
            yield portal
            await portal.cancel_actor()
 async def spawn_brokerd(
    brokername: str,
    loglevel: Optional[str] = None,
    **tractor_kwargs,
 ) -> bool:
    log.info(f'Spawning {brokername} broker daemon')
    brokermod = get_brokermod(brokername)
    dname = f'brokerd.{brokername}'
    extra_tractor_kwargs = getattr(brokermod, '_spawn_kwargs', {})
    tractor_kwargs.update(extra_tractor_kwargs)
    # ask `pikerd` to spawn a new sub-actor and manage it under its
    # actor nursery
    modpath = brokermod.__name__
    broker_enable = [modpath]
    for submodname in getattr(
        brokermod,
        '__enable_modules__',
        [],
    ):
        subpath = f'{modpath}.{submodname}'
        broker_enable.append(subpath)
    portal = await Services.actor_n.start_actor(
        dname,
        enable_modules=_data_mods + broker_enable,
        loglevel=loglevel,
        debug_mode=Services.debug_mode,
        **tractor_kwargs
    )
    # non-blocking setup of brokerd service nursery
    from .data import _setup_persistent_brokerd
    await Services.start_service_task(
        dname,
        portal,
        _setup_persistent_brokerd,
        brokername=brokername,
    )
    return True
@acm
 async def maybe_spawn_brokerd(
    brokername: str,
    loglevel: Optional[str] = None,
    **kwargs,
 ) -> tractor.Portal:
    '''
    Helper to spawn a brokerd service *from* a client
    who wishes to use the sub-actor-daemon.
    '''
    async with maybe_spawn_daemon(
        f'brokerd.{brokername}',
        service_task_target=spawn_brokerd,
        spawn_args={'brokername': brokername, 'loglevel': loglevel},
        loglevel=loglevel,
        **kwargs,
    ) as portal:
        yield portal
 async def spawn_emsd(
    loglevel: Optional[str] = None,
    **extra_tractor_kwargs
 ) -> bool:
    """
    Start the clearing engine under ``pikerd``.
    """
    log.info('Spawning emsd')
    portal = await Services.actor_n.start_actor(
        'emsd',
        enable_modules=[
            'piker.clearing._ems',
            'piker.clearing._client',
        ],
        loglevel=loglevel,
        debug_mode=Services.debug_mode,  # set by pikerd flag
        **extra_tractor_kwargs
    )
    # non-blocking setup of clearing service
    from .clearing._ems import _setup_persistent_emsd
    await Services.start_service_task(
        'emsd',
        portal,
        _setup_persistent_emsd,
    )
    return True
@acm
 async def maybe_open_emsd(
    brokername: str,
    loglevel: Optional[str] = None,
    **kwargs,
 ) -> tractor._portal.Portal:  # noqa
    async with maybe_spawn_daemon(
        'emsd',
        service_task_target=spawn_emsd,
        spawn_args={'loglevel': loglevel},
        loglevel=loglevel,
        **kwargs,
    ) as portal:
        yield portal
--- a/piker/toolz/profile.py
+++ b/piker/toolz/profile.py
@ -152,14 +152,9 @@ class Profiler(object):
            # don't do anything
            return cls._disabledProfiler
        # create an actual profiling object
        cls._depth += 1
        obj = super(Profiler, cls).__new__(cls)
        obj._msgs = []
        # create an actual profiling object
        if cls._depth < 1:
            cls._msgs = []
        obj._name = msg or func_qualname
        obj._delayed = delayed
        obj._markCount = 0
@ -179,12 +174,8 @@ class Profiler(object):
        self._markCount += 1
        newTime = perf_counter()
        tot_ms = (newTime - self._firstTime) * 1000
        ms = (newTime - self._lastTime) * 1000
-        self._newMsg(
+        self._newMsg("  %s: %0.4f ms", msg, ms)
            f"  {msg}: {ms:0.4f}, tot:{tot_ms:0.4f}"
        )
        self._lastTime = newTime
    def mark(self, msg=None):
--- a/piker/accounting/README.rst
+++ b/piker/accounting/README.rst
@ -1,16 +0,0 @@
 .accounting
 -----------
 A subsystem for transaction processing, storage and historical
 measurement.
 .pnl
 ----
 BEP, the break even price: the price at which liquidating
 a remaining position results in a zero PnL since the position was
 "opened" in the destination asset.
 PPU: price-per-unit: the "average cost" (in cumulative mean terms)
 of the "entry" transactions which "make a position larger"; taking
 a profit relative to this price means that you will "make more
 profit then made prior" since the position was opened.
--- a/piker/accounting/init.py
+++ b/piker/accounting/init.py
@ -1,115 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 "Accounting for degens": count dem numberz that tracks how much you got
 for tendiez.
 '''
 from piker.log import (
    get_console_log,
    get_logger,
 )
 from .calc import (
    iter_by_dt,
 )
 from ._ledger import (
    Transaction,
    TransactionLedger,
    open_trade_ledger,
 )
 from ._pos import (
    Account,
    load_account,
    load_account_from_ledger,
    open_account,
    Position,
 )
 from ._mktinfo import (
    Asset,
    dec_digits,
    digits_to_dec,
    MktPair,
    unpack_fqme,
    _derivs as DerivTypes,
 )
 from ._allocate import (
    mk_allocator,
    Allocator,
 )
 log = get_logger(__name__)
 # ?TODO, enable console on import
 # [ ] necessary? or `open_brokerd_dialog()` doing it is sufficient?
 #
 # bc might as well enable whenev imported by
 # other sub-sys code (namely `.clearing`).
 get_console_log(
    level='warning',
    name=__name__,
 )
 # TODO, the `as <samename>` style?
 __all__ = [
    'Account',
    'Allocator',
    'Asset',
    'MktPair',
    'Position',
    'Transaction',
    'TransactionLedger',
    'dec_digits',
    'digits_to_dec',
    'iter_by_dt',
    'load_account',
    'load_account_from_ledger',
    'mk_allocator',
    'open_account',
    'open_trade_ledger',
    'unpack_fqme',
    'DerivTypes',
 ]
 def get_likely_pair(
    src: str,
    dst: str,
    bs_mktid: str,
 ) -> str | None:
    '''
    Attempt to get the likely trading pair matching a given destination
    asset `dst: str`.
    '''
    try:
        src_name_start: str = bs_mktid.rindex(src)
    except (
        ValueError,   # substr not found
    ):
        # TODO: handle nested positions..(i.e.
        # positions where the src fiat was used to
        # buy some other dst which was furhter used
        # to buy another dst..)
        # log.warning(
        #     f'No src fiat {src} found in {bs_mktid}?'
        # )
        return None
    likely_dst: str = bs_mktid[:src_name_start]
    if likely_dst == dst:
        return bs_mktid
--- a/piker/accounting/_ledger.py
+++ b/piker/accounting/_ledger.py
@ -1,429 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Trade and transaction ledger processing.
 '''
 from __future__ import annotations
 from collections import UserDict
 from contextlib import contextmanager as cm
 from functools import partial
 from pathlib import Path
 from pprint import pformat
 from types import ModuleType
 from typing import (
    Any,
    Callable,
    Generator,
    Literal,
    TYPE_CHECKING,
 )
 from pendulum import (
    DateTime,
 )
 import tomli_w  # for fast ledger writing
 from piker.types import Struct
 from piker import config
 from piker.log import get_logger
 from .calc import (
    iter_by_dt,
 )
 if TYPE_CHECKING:
    from ..data._symcache import (
        SymbologyCache,
    )
 log = get_logger(__name__)
 TxnType = Literal[
    'clear',
    'transfer',
    # TODO: see https://github.com/pikers/piker/issues/510
    # 'split',
    # 'rename',
    # 'resize',
    # 'removal',
 ]
 class Transaction(Struct, frozen=True):
    # NOTE: this is a unified acronym also used in our `MktPair`
    # and can stand for any of a
    # "fully qualified <blank> endpoint":
    # - "market" in the case of financial trades
    #   (btcusdt.spot.binance).
    # - "merkel (tree)" aka a blockchain system "wallet tranfers"
    #   (btc.blockchain)
    # - "money" for tradtitional (digital databases)
    #   *bank accounts* (usd.swift, eur.sepa)
    fqme: str
    tid: str | int  # unique transaction id
    size: float
    price: float
    cost: float  # commisions or other additional costs
    dt: DateTime
    # the "event type" in terms of "market events" see above and
    # https://github.com/pikers/piker/issues/510
    etype: TxnType = 'clear'
    # TODO: we can drop this right since we
    # can instead expect the backend to provide this
    # via the `MktPair`?
    expiry: DateTime | None = None
    # (optional) key-id defined by the broker-service backend which
    # ensures the instrument-symbol market key for this record is unique
    # in the "their backend/system" sense; i.e. this uid for the market
    # as defined (internally) in some namespace defined by the broker
    # service.
    bs_mktid: str | int | None = None
    def to_dict(
        self,
        **kwargs,
    ) -> dict:
        dct: dict[str, Any] = super().to_dict(**kwargs)
        # ensure we use a pendulum formatted
        # ISO style str here!@
        dct['dt'] = str(self.dt)
        return dct
 class TransactionLedger(UserDict):
    '''
    Very simple ``dict`` wrapper + ``pathlib.Path`` handle to
    a TOML formatted transaction file for enabling file writes
    dynamically whilst still looking exactly like a ``dict`` from the
    outside.
    '''
    # NOTE: see `open_trade_ledger()` for defaults, this should
    # never be constructed manually!
    def __init__(
        self,
        ledger_dict: dict,
        file_path: Path,
        account: str,
        mod: ModuleType,  # broker mod
        tx_sort: Callable,
        symcache: SymbologyCache,
    ) -> None:
        self.account: str = account
        self.file_path: Path = file_path
        self.mod: ModuleType = mod
        self.tx_sort: Callable = tx_sort
        self._symcache: SymbologyCache = symcache
        # any added txns we keep in that form for meta-data
        # gathering purposes
        self._txns: dict[str, Transaction] = {}
        super().__init__(ledger_dict)
    def __repr__(self) -> str:
        return (
            f'TransactionLedger: {len(self)}\n'
            f'{pformat(list(self.data))}'
        )
    @property
    def symcache(self) -> SymbologyCache:
        '''
        Read-only ref to backend's ``SymbologyCache``.
        '''
        return self._symcache
    def update_from_t(
        self,
        t: Transaction,
    ) -> None:
        '''
        Given an input `Transaction`, cast to `dict` and update
        from it's transaction id.
        '''
        self.data[t.tid] = t.to_dict()
        self._txns[t.tid] = t
    def iter_txns(
        self,
        symcache: SymbologyCache | None = None,
    ) -> Generator[
        Transaction,
        None,
        None,
    ]:
        '''
        Deliver trades records in ``(key: str, t: Transaction)``
        form via generator.
        '''
        symcache = symcache or self._symcache
        if self.account == 'paper':
            from piker.clearing import _paper_engine
            norm_trade: Callable = partial(
                _paper_engine.norm_trade,
                brokermod=self.mod,
            )
        else:
            norm_trade: Callable = self.mod.norm_trade
        # datetime-sort and pack into txs
        for tid, txdict in self.tx_sort(self.data.items()):
            txn: Transaction = norm_trade(
                tid,
                txdict,
                pairs=symcache.pairs,
                symcache=symcache,
            )
            yield txn
    def to_txns(
        self,
        symcache: SymbologyCache | None = None,
    ) -> dict[str, Transaction]:
        '''
        Return entire output from ``.iter_txns()`` in a ``dict``.
        '''
        txns: dict[str, Transaction] = {}
        for t in self.iter_txns(symcache=symcache):
            if not t:
                log.warning(f'{self.mod.name}:{self.account} TXN is -> {t}')
                continue
            txns[t.tid] = t
        return txns
    def write_config(self) -> None:
        '''
        Render the self.data ledger dict to its TOML file form.
        ALWAYS order datetime sorted!
        '''
        is_paper: bool = self.account == 'paper'
        symcache: SymbologyCache = self._symcache
        towrite: dict[str, Any] = {}
        for tid, txdict in self.tx_sort(
            self.data.copy()
        ):
            # write blank-str expiry for non-expiring assets
            if (
                'expiry' in txdict
                and txdict['expiry'] is None
            ):
                txdict['expiry'] = ''
            # (maybe) re-write old acro-key
            if (
                is_paper
                # if symcache is empty/not supported (yet), don't
                # bother xD
                and symcache.mktmaps
            ):
                fqme: str = txdict.pop('fqsn', None) or txdict['fqme']
                bs_mktid: str | None = txdict.get('bs_mktid')
                if (
                    fqme not in symcache.mktmaps
                    or (
                        # also try to see if this is maybe a paper
                        # engine ledger in which case the bs_mktid
                        # should be the fqme as well!
                        bs_mktid
                        and fqme != bs_mktid
                    )
                ):
                    # always take any (paper) bs_mktid if defined and
                    # in the backend's cache key set.
                    if bs_mktid in symcache.mktmaps:
                        fqme: str = bs_mktid
                    else:
                        best_fqme: str = list(symcache.search(fqme))[0]
                        log.warning(
                            f'Could not find FQME: {fqme} in qualified set?\n'
                            f'Qualifying and expanding {fqme} -> {best_fqme}'
                        )
                        fqme = best_fqme
                if (
                    bs_mktid
                    and bs_mktid != fqme
                ):
                    # in paper account case always make sure both the
                    # fqme and bs_mktid are fully qualified..
                    txdict['bs_mktid'] = fqme
                # in paper ledgers always write the latest
                # symbology key field: an FQME.
                txdict['fqme'] = fqme
            towrite[tid] = txdict
        with self.file_path.open(mode='wb') as fp:
            tomli_w.dump(towrite, fp)
 def load_ledger(
    brokername: str,
    acctid: str,
    # for testing or manual load from file
    dirpath: Path | None = None,
 ) -> tuple[dict, Path]:
    '''
    Load a ledger (TOML) file from user's config directory:
    $CONFIG_DIR/accounting/ledgers/trades_<brokername>_<acctid>.toml
    Return its `dict`-content and file path.
    '''
    import time
    try:
        import tomllib
    except ModuleNotFoundError:
        import tomli as tomllib
    ldir: Path = (
        dirpath
        or
        config._config_dir / 'accounting' / 'ledgers'
    )
    if not ldir.is_dir():
        ldir.mkdir()
    fname = f'trades_{brokername}_{acctid}.toml'
    fpath: Path = ldir / fname
    if not fpath.is_file():
        log.info(
            f'Creating new local trades ledger: {fpath}'
        )
        fpath.touch()
    with fpath.open(mode='rb') as cf:
        start = time.time()
        ledger_dict = tomllib.load(cf)
        log.debug(f'Ledger load took {time.time() - start}s')
    return ledger_dict, fpath
@cm
 def open_trade_ledger(
    broker: str,
    account: str,
    allow_from_sync_code: bool = False,
    symcache: SymbologyCache | None = None,
    # default is to sort by detected datetime-ish field
    tx_sort: Callable = iter_by_dt,
    rewrite: bool = False,
    # for testing or manual load from file
    _fp: Path | None = None,
 ) -> Generator[TransactionLedger, None, None]:
    '''
    Indempotently create and read in a trade log file from the
    ``<configuration_dir>/ledgers/`` directory.
    Files are named per broker account of the form
    ``<brokername>_<accountname>.toml``. The ``accountname`` here is the
    name as defined in the user's ``brokers.toml`` config.
    '''
    from ..brokers import get_brokermod
    mod: ModuleType = get_brokermod(broker)
    ledger_dict, fpath = load_ledger(
        broker,
        account,
        dirpath=_fp,
    )
    cpy: dict = ledger_dict.copy()
    # XXX NOTE: if not provided presume we are being called from
    # sync code and need to maybe run `trio` to generate..
    if symcache is None:
        # XXX: be mega pendantic and ensure the caller knows what
        # they're doing!
        if not allow_from_sync_code:
            raise RuntimeError(
                'You MUST set `allow_from_sync_code=True` when '
                'calling `open_trade_ledger()` from sync code! '
                'If you are calling from async code you MUST '
                'instead pass a `symcache: SymbologyCache`!'
            )
        from ..data._symcache import (
            get_symcache,
        )
        symcache: SymbologyCache = get_symcache(broker)
    assert symcache
    ledger = TransactionLedger(
        ledger_dict=cpy,
        file_path=fpath,
        account=account,
        mod=mod,
        symcache=symcache,
        # NOTE: allow backends to provide custom ledger sorting
        tx_sort=getattr(
            mod,
            'tx_sort',
            tx_sort,
        ),
    )
    try:
        yield ledger
    finally:
        if (
            ledger.data != ledger_dict
            or rewrite
        ):
            # TODO: show diff output?
            # https://stackoverflow.com/questions/12956957/print-diff-of-python-dictionaries
            log.info(f'Updating ledger for {fpath}:\n')
            ledger.write_config()
--- a/piker/accounting/_mktinfo.py
+++ b/piker/accounting/_mktinfo.py
@ -1,679 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Market (pair) meta-info layer: sane addressing semantics and meta-data
 for cross-provider marketplaces.
 We intoduce the concept of,
 - a FQMA: fully qualified market address,
 - a sane schema for FQMAs including derivatives,
 - a msg-serializeable description of markets for
  easy sharing with other pikers B)
 '''
 from __future__ import annotations
 from decimal import (
    Decimal,
    ROUND_HALF_EVEN,
 )
 from typing import (
    Any,
    Literal,
 )
 from piker.types import Struct
 # TODO: make these literals..
 _underlyings: list[str] = [
    'stock',
    'bond',
    'crypto',
    'fiat',
    'commodity',
 ]
 _crypto_derivs: list[str] = [
    'perpetual_future',
    'crypto_future',
 ]
 _derivs: list[str] = [
    'swap',
    'future',
    'continuous_future',
    'option',
    'futures_option',
    # if we can't figure it out, presume the worst XD
    'unknown',
 ]
 # NOTE: a tag for other subsystems to try
 # and do default settings for certain things:
 # - allocator does unit vs. dolla size limiting.
 AssetTypeName: Literal[
    _underlyings
    +
    _derivs
    +
    _crypto_derivs
 ]
 # egs. stock, futer, option, bond etc.
 def dec_digits(
    value: float | str | Decimal,
 ) -> int:
    '''
    Return the number of precision digits read from a decimal or float
    value.
    '''
    if value == 0:
        return 0
    return int(
        -Decimal(str(value)).as_tuple().exponent
    )
 float_digits = dec_digits
 def digits_to_dec(
    ndigits: int,
 ) -> Decimal:
    '''
    Return the minimum float value for an input integer value.
    eg. 3 -> 0.001
    '''
    if ndigits == 0:
        return Decimal('0')
    return Decimal('0.' + '0'*(ndigits-1) + '1')
 class Asset(Struct, frozen=True):
    '''
    Container type describing any transactable asset and its
    contract-like and/or underlying technology meta-info.
    '''
    name: str
    atype: str  # AssetTypeName
    # minimum transaction size / precision.
    # eg. for buttcoin this is a "satoshi".
    tx_tick: Decimal
    # NOTE: additional info optionally packed in by the backend, but
    # should not be explicitly required in our generic API.
    info: dict | None = None
    # `None` is not toml-compat so drop info
    # if no extra data added..
    def to_dict(
        self,
        **kwargs,
    ) -> dict:
        dct = super().to_dict(**kwargs)
        if (info := dct.pop('info', None)):
            dct['info'] = info
        assert dct['tx_tick']
        return dct
    @classmethod
    def from_msg(
        cls,
        msg: dict[str, Any],
    ) -> Asset:
        return cls(
            tx_tick=Decimal(str(msg.pop('tx_tick'))),
            info=msg.pop('info', None),
            **msg,
        )
    def __str__(self) -> str:
        return self.name
    def quantize(
        self,
        size: float,
    ) -> Decimal:
        '''
        Truncate input ``size: float`` using ``Decimal``
        quantized form of the digit precision defined
        by ``self.lot_tick_size``.
        '''
        digits = float_digits(self.tx_tick)
        return Decimal(size).quantize(
            Decimal(f'1.{"0".ljust(digits, "0")}'),
            rounding=ROUND_HALF_EVEN
        )
    @classmethod
    def guess_from_mkt_ep_key(
        cls,
        mkt_ep_key: str,
        atype: str | None = None,
    ) -> Asset:
        '''
        A hacky guess method for presuming a (target) asset's properties
        based on either the actualy market endpoint key, or config settings
        from the user.
        '''
        atype = atype or 'unknown'
        # attempt to strip off any source asset
        # via presumed syntax of:
        # - <dst>/<src>
        # - <dst>.<src>
        # - etc.
        for char in ['/', '.']:
            dst, _, src = mkt_ep_key.partition(char)
            if src:
                if not atype:
                    atype = 'fiat'
                break
        return Asset(
            name=dst,
            atype=atype,
            tx_tick=Decimal('0.01'),
        )
 def maybe_cons_tokens(
    tokens: list[Any],
    delim_char: str = '.',
 ) -> str:
    '''
    Construct `str` output from a maybe-concatenation of input
    sequence of elements in ``tokens``.
    '''
    return delim_char.join(filter(bool, tokens)).lower()
 class MktPair(Struct, frozen=True):
    '''
    Market description for a pair of assets which are tradeable:
    a market which enables transactions of the form,
        buy: source asset -> destination asset
        sell: destination asset -> source asset
    The main intention of this type is for a **simple** cross-asset
    venue/broker normalized descrption type from which all
    market-auctions can be mapped from FQME identifiers.
    TODO: our eventual target fqme format/schema is:
    <dst>/<src>.<expiry>.<con_info_1>.<con_info_2>. -> .<venue>.<broker>
          ^ -- optional tokens ------------------------------- ^
    Notes:
    ------
    Some venues provide a different semantic (which we frankly find
    confusing and non-general) such as "base" and "quote" asset.
    For example this is how `binance` defines the terms:
    https://binance-docs.github.io/apidocs/websocket_api/en/#public-api-definitions
    https://binance-docs.github.io/apidocs/futures/en/#public-endpoints-info
    - *base* asset refers to the asset that is the *quantity* of a symbol.
    - *quote* asset refers to the asset that is the *price* of a symbol.
    In other words the "quote" asset is the asset that the market
    is pricing "buys" *in*, and the *base* asset it the one that the market
    allows you to "buy" an *amount of*. Put more simply the *quote*
    asset is our "source" asset and the *base* asset is our "destination"
    asset.
    This defintion can be further understood reading our
    `.brokers.binance.api.Pair` type wherein the
    `Pair.[quote/base]AssetPrecision` field determines the (transfer)
    transaction precision available per asset; i.e. the satoshis
    unit in bitcoin for representing the minimum size of a
    transaction that can take place on the blockchain.
    '''
    dst: str | Asset
    # "destination asset" (name) used to buy *to*
    # (or used to sell *from*)
    price_tick: Decimal  # minimum price increment
    size_tick: Decimal  # minimum size (aka vlm) increment
    # the tick size is the number describing the smallest step in value
    # available in this market between the source and destination
    # assets.
    # https://en.wikipedia.org/wiki/Tick_size
    # https://en.wikipedia.org/wiki/Commodity_tick
    # https://en.wikipedia.org/wiki/Percentage_in_point
    # unique "broker id" since every market endpoint provider
    # has their own nomenclature and schema for market maps.
    bs_mktid: str
    broker: str  # the middle man giving access
    # NOTE: to start this field is optional but should eventually be
    # required; the reason is for backward compat since more positioning
    # calculations were not originally stored with a src asset..
    src: str | Asset = ''
    # "source asset" (name) used to buy *from*
    # (or used to sell *to*).
    venue: str = ''  # market venue provider name
    expiry: str = ''  # for derivs, expiry datetime parseable str
    # destination asset's financial type/classification name
    # NOTE: this is required for the order size allocator system,
    # since we use different default settings based on the type
    # of the destination asset, eg. futes use a units limits vs.
    # equities a $limit.
    # dst_type: AssetTypeName | None = None
    # source asset's financial type/classification name
    # TODO: is a src type required for trading?
    # there's no reason to need any more then the one-way alloc-limiter
    # config right?
    # src_type: AssetTypeName
    # for derivs, info describing contract, egs. strike price, call
    # or put, swap type, exercise model, etc.
    contract_info: list[str] | None = None
    # TODO: rename to sectype since all of these can
    # be considered "securities"?
    _atype: str = ''
    # allow explicit disable of the src part of the market
    # pair name -> useful for legacy markets like qqq.nasdaq.ib
    _fqme_without_src: bool = False
    # NOTE: when cast to `str` return fqme
    def __str__(self) -> str:
        return self.fqme
    def to_dict(
        self,
        **kwargs,
    ) -> dict:
        d = super().to_dict(**kwargs)
        d['src'] = self.src.to_dict(**kwargs)
        if not isinstance(self.dst, str):
            d['dst'] = self.dst.to_dict(**kwargs)
        else:
            d['dst'] = str(self.dst)
        d['price_tick'] = str(self.price_tick)
        d['size_tick'] = str(self.size_tick)
        if self.contract_info is None:
            d.pop('contract_info')
        # d.pop('_fqme_without_src')
        return d
    @classmethod
    def from_msg(
        cls,
        msg: dict[str, Any],
    ) -> MktPair:
        '''
        Constructor for a received msg-dict normally received over IPC.
        '''
        if not isinstance(
            dst_asset_msg := msg.pop('dst'),
            str,
        ):
            dst: Asset = Asset.from_msg(dst_asset_msg)  # .copy()
        else:
            dst: str = dst_asset_msg
        src_asset_msg: dict = msg.pop('src')
        src: Asset = Asset.from_msg(src_asset_msg)  # .copy()
        # XXX NOTE: ``msgspec`` can encode `Decimal` but it doesn't
        # decide to it by default since we aren't spec-cing these
        # msgs as structs proper to get them to decode implictily
        # (yet) as per,
        # - https://github.com/pikers/piker/pull/354
        # - https://github.com/goodboy/tractor/pull/311
        # SO we have to ensure we do a struct type
        # case (which `.copy()` does) to ensure we get the right
        # type!
        return cls(
            dst=dst,
            src=src,
            price_tick=Decimal(msg.pop('price_tick')),
            size_tick=Decimal(msg.pop('size_tick')),
            **msg,
        ).copy()
    @property
    def resolved(self) -> bool:
        return isinstance(self.dst, Asset)
    @classmethod
    def from_fqme(
        cls,
        fqme: str,
        price_tick: float|str,
        size_tick: float|str,
        bs_mktid: str,
        broker: str | None = None,
        **kwargs,
    ) -> MktPair:
        _fqme: str = fqme
        if (
            broker
            and broker not in fqme
        ):
            _fqme = f'{fqme}.{broker}'
        broker, mkt_ep_key, venue, expiry = unpack_fqme(_fqme)
        kven: str = kwargs.pop('venue', venue)
        if venue:
            assert venue == kven
        else:
            venue = kven
        exp: str = kwargs.pop('expiry', expiry)
        if expiry:
            assert exp == expiry
        else:
            expiry = exp
        dst: Asset = Asset.guess_from_mkt_ep_key(
            mkt_ep_key,
            atype=kwargs.get('_atype'),
        )
        # XXX: loading from a fqme string will
        # leave this pair as "un resolved" meaning
        # we don't yet have `.dst` set as an `Asset`
        # which we expect to be filled in by some
        # backend client with access to that data-info.
        return cls(
            dst=dst,
            # XXX: not resolved to ``Asset`` :(
            #src=src,
            broker=broker,
            venue=venue,
            # XXX NOTE: we presume this token
            # if the expiry for now!
            expiry=expiry,
            price_tick=price_tick,
            size_tick=size_tick,
            bs_mktid=bs_mktid,
            **kwargs,
        ).copy()
    @property
    def key(self) -> str:
        '''
        The "endpoint key" for this market.
        '''
        return self.pair
    def pair(
        self,
        delim_char: str | None = None,
    ) -> str:
        '''
        The "endpoint asset pair key" for this market.
        Eg. mnq/usd or btc/usdt or xmr/btc
        In most other tina platforms this is referred to as the
        "symbol".
        '''
        return maybe_cons_tokens(
            [str(self.dst),
             str(self.src)],
            # TODO: make the default '/'
            delim_char=delim_char or '',
        )
    @property
    def suffix(self) -> str:
        '''
        The "contract suffix" for this market.
        Eg. mnq/usd.20230616.cme.ib
                    ^ ----- ^
        or tsla/usd.20230324.200c.cboe.ib
                    ^ ---------- ^
        In most other tina platforms they only show you these details in
        some kinda "meta data" format, we have FQMEs so we do this up
        front and explicit.
        '''
        field_strs = [self.expiry]
        con_info = self.contract_info
        if con_info is not None:
            field_strs.extend(con_info)
        return maybe_cons_tokens(field_strs)
    def get_fqme(
        self,
        # NOTE: allow dropping the source asset from the
        # market endpoint's pair key. Eg. to change
        # mnq/usd.<> -> mnq.<> which is useful when
        # searching (legacy) stock exchanges.
        without_src: bool = False,
        delim_char: str | None = None,
    ) -> str:
        '''
        Return the fully qualified market endpoint-address for the
        pair of transacting assets.
        fqme = "fully qualified market endpoint"
        And yes, you pronounce it colloquially as read..
        Basically the idea here is for all client code (consumers of piker's
        APIs which query the data/broker-provider agnostic layer(s)) should be
        able to tell which backend / venue / derivative each data feed/flow is
        from by an explicit string-key of the current form:
        <market-instrument-name>
            .<venue>
            .<expiry>
            .<derivative-suffix-info>
            .<brokerbackendname>
        eg. for an explicit daq mini futes contract: mnq.cme.20230317.ib
        TODO: I have thoughts that we should actually change this to be
        more like an "attr lookup" (like how the web should have done
        urls, but marketting peeps ruined it etc. etc.)
        <broker>.<venue>.<instrumentname>.<suffixwithmetadata>
        TODO:
        See community discussion on naming and nomenclature, order
        of addressing hierarchy, general schema, internal representation:
        https://github.com/pikers/piker/issues/467
        '''
        key: str = (
            self.pair(delim_char=delim_char)
            if not (without_src or self._fqme_without_src)
            else str(self.dst)
        )
        return maybe_cons_tokens([
            key,  # final "pair name" (eg. qqq[/usd], btcusdt)
            self.venue,
            self.suffix,  # includes expiry and other con info
            self.broker,
        ])
    # NOTE: the main idea behind an fqme is to map a "market address"
    # to some endpoint from a transaction provider (eg. a broker) such
    # that we build a table of `fqme: str -> bs_mktid: Any` where any "piker
    # market address" maps 1-to-1 to some broker trading endpoint.
    # @cached_property
    fqme = property(get_fqme)
    def get_bs_fqme(
        self,
        **kwargs,
    ) -> str:
        '''
        FQME sin broker part XD
        '''
        sin_broker, *_ = self.get_fqme(**kwargs).rpartition('.')
        return sin_broker
    bs_fqme = property(get_bs_fqme)
    @property
    def fqsn(self) -> str:
        return self.fqme
    def quantize(
        self,
        size: float,
        quantity_type: Literal['price', 'size'] = 'size',
    ) -> Decimal:
        '''
        Truncate input ``size: float`` using ``Decimal``
        and ``.size_tick``'s # of digits.
        '''
        match quantity_type:
            case 'price':
                digits = float_digits(self.price_tick)
            case 'size':
                digits = float_digits(self.size_tick)
        return Decimal(size).quantize(
            Decimal(f'1.{"0".ljust(digits, "0")}'),
            rounding=ROUND_HALF_EVEN
        )
    # TODO: BACKWARD COMPAT, TO REMOVE?
    @property
    def type_key(self) -> str:
        # if set explicitly then use it!
        if self._atype:
            return self._atype
        if isinstance(self.dst, Asset):
            return str(self.dst.atype)
        return 'UNKNOWN'
    @property
    def price_tick_digits(self) -> int:
        return float_digits(self.price_tick)
    @property
    def size_tick_digits(self) -> int:
        return float_digits(self.size_tick)
 def unpack_fqme(
    fqme: str,
    broker: str | None = None
 ) -> tuple[str, ...]:
    '''
    Unpack a fully-qualified-symbol-name to ``tuple``.
    '''
    venue = ''
    suffix = ''
    # TODO: probably reverse the order of all this XD
    tokens = fqme.split('.')
    match tokens:
        case [mkt_ep, broker]:
            # probably crypto
            return (
                broker,
                mkt_ep,
                '',
                '',
            )
        # TODO: swap venue and suffix/deriv-info here?
        case [mkt_ep, venue, suffix, broker]:
            pass
        # handle `bs_mktid` + `broker` input case
        case [
            mkt_ep, venue, suffix
        ] if (
            broker
            and suffix != broker
        ):
            pass
        case [mkt_ep, venue, broker]:
            suffix = ''
        case _:
            raise ValueError(f'Invalid fqme: {fqme}')
    return (
        broker,
        mkt_ep,
        venue,
        # '.'.join([mkt_ep, venue]),
        suffix,
    )
--- a/piker/accounting/_pos.py
+++ b/piker/accounting/_pos.py
--- a/piker/accounting/calc.py
+++ b/piker/accounting/calc.py
@ -1,768 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Calculation routines for balance and position tracking such that
 you know when you're losing money (if possible) XD
 '''
 from __future__ import annotations
 from collections.abc import ValuesView
 from contextlib import contextmanager as cm
 from functools import partial
 from math import copysign
 from pprint import pformat
 from typing import (
    Any,
    Callable,
    Iterator,
    TYPE_CHECKING,
 )
 from tractor.devx import maybe_open_crash_handler
 import polars as pl
 from pendulum import (
    DateTime,
    from_timestamp,
    parse,
 )
 from ..log import get_logger
 if TYPE_CHECKING:
    from ._ledger import (
        Transaction,
        TransactionLedger,
    )
 log = get_logger(__name__)
 def ppu(
    clears: Iterator[Transaction],
    # include transaction cost in breakeven price
    # and presume the worst case of the same cost
    # to exit this transaction (even though in reality
    # it will be dynamic based on exit stratetgy).
    cost_scalar: float = 2,
    # return the ledger of clears as a (now dt sorted) dict with
    # new position fields inserted alongside each entry.
    as_ledger: bool = False,
 ) -> float | list[(str, dict)]:
    '''
    Compute the "price-per-unit" price for the given non-zero sized
    rolling position.
    The recurrence relation which computes this (exponential) mean
    per new clear which **increases** the accumulative postiion size
    is:
    ppu[-1] = (
        ppu[-2] * accum_size[-2]
        +
        ppu[-1] * size
    ) / accum_size[-1]
    where `cost_basis` for the current step is simply the price
    * size of the most recent clearing transaction.
    -----
    TODO: get the BEP computed and working similarly!
    -----
    the equivalent "break even price" or bep at each new clear
    event step conversely only changes when an "position exiting
    clear" which **decreases** the cumulative dst asset size:
    bep[-1] = ppu[-1] - (cum_pnl[-1] / cumsize[-1])
    '''
    asize_h: list[float] = []  # historical accumulative size
    ppu_h: list[float] = []  # historical price-per-unit
    # ledger: dict[str, dict] = {}
    ledger: list[dict] = []
    t: Transaction
    for t in clears:
        clear_size: float = t.size
        clear_price: str | float = t.price
        is_clear: bool = not isinstance(clear_price, str)
        last_accum_size = asize_h[-1] if asize_h else 0
        accum_size: float = last_accum_size + clear_size
        accum_sign = copysign(1, accum_size)
        sign_change: bool = False
        # on transfers we normally write some non-valid
        # price since withdrawal to another account/wallet
        # has nothing to do with inter-asset-market prices.
        # TODO: this should be better handled via a `type: 'tx'`
        # field as per existing issue surrounding all this:
        # https://github.com/pikers/piker/issues/510
        if isinstance(clear_price, str):
            # TODO: we can't necessarily have this commit to
            # the overall pos size since we also need to
            # include other positions contributions to this
            # balance or we might end up with a -ve balance for
            # the position..
            continue
        # test if the pp somehow went "passed" a net zero size state
        # resulting in a change of the "sign" of the size (+ve for
        # long, -ve for short).
        sign_change = (
            copysign(1, last_accum_size) + accum_sign == 0
            and last_accum_size != 0
        )
        # since we passed the net-zero-size state the new size
        # after sum should be the remaining size the new
        # "direction" (aka, long vs. short) for this clear.
        if sign_change:
            clear_size: float = accum_size
            abs_diff: float = abs(accum_size)
            asize_h.append(0)
            ppu_h.append(0)
        else:
            # old size minus the new size gives us size diff with
            # +ve -> increase in pp size
            # -ve -> decrease in pp size
            abs_diff = abs(accum_size) - abs(last_accum_size)
        # XXX: LIFO breakeven price update. only an increaze in size
        # of the position contributes the breakeven price,
        # a decrease does not (i.e. the position is being made
        # smaller).
        # abs_clear_size = abs(clear_size)
        abs_new_size: float | int = abs(accum_size)
        if (
            abs_diff > 0
            and is_clear
        ):
            cost_basis = (
                # cost basis for this clear
                clear_price * abs(clear_size)
                +
                # transaction cost
                accum_sign * cost_scalar * t.cost
            )
            if asize_h:
                size_last: float = abs(asize_h[-1])
                cb_last: float = ppu_h[-1] * size_last
                ppu: float = (cost_basis + cb_last) / abs_new_size
            else:
                ppu: float = cost_basis / abs_new_size
        else:
            # TODO: for PPU we should probably handle txs out
            # (aka withdrawals) similarly by simply not having
            # them contrib to the running PPU calc and only
            # when the next entry clear comes in (which will
            # then have a higher weighting on the PPU).
            # on "exit" clears from a given direction,
            # only the size changes not the price-per-unit
            # need to be updated since the ppu remains constant
            # and gets weighted by the new size.
            ppu: float = ppu_h[-1] if ppu_h else 0  # set to previous value
        # extend with new rolling metric for this step
        ppu_h.append(ppu)
        asize_h.append(accum_size)
        # ledger[t.tid] = {
            # 'txn': t,
        # ledger[t.tid] = t.to_dict() | {
        ledger.append((
            t.tid,
            t.to_dict() | {
                'ppu': ppu,
                'cumsize': accum_size,
                'sign_change': sign_change,
                # TODO: cum_pnl, bep
            }
        ))
    final_ppu = ppu_h[-1] if ppu_h else 0
    # TODO: once we have etypes in all ledger entries..
    # handle any split info entered (for now) manually by user
    # if self.split_ratio is not None:
    #     final_ppu /= self.split_ratio
    if as_ledger:
        return ledger
    else:
        return final_ppu
 def iter_by_dt(
    records: (
        dict[str, dict[str, Any]]
        | ValuesView[dict]  # eg. `Position._events.values()`
        | list[dict]
        | list[Transaction]  # XXX preferred!
    ),
    # NOTE: parsers are looked up in the insert order
    # so if you know that the record stats show some field
    # is more common then others, stick it at the top B)
    parsers: dict[str, Callable | None] = {
        'dt': parse,  # parity case
        'datetime': parse,  # datetime-str
        'time': from_timestamp,  # float epoch
    },
    key: Callable | None = None,
 ) -> Iterator[tuple[str, dict]]:
    '''
    Iterate entries of a transaction table sorted by entry recorded
    datetime presumably set at the ``'dt'`` field in each entry.
    '''
    if isinstance(records, dict):
        records: list[tuple[str, dict]] = list(records.items())
    def dyn_parse_to_dt(
        tx: tuple[str, dict[str, Any]] | Transaction,
        debug: bool = False,
        _invalid: list|None = None,
    ) -> DateTime:
        # handle `.items()` inputs
        if isinstance(tx, tuple):
            tx = tx[1]
        # dict or tx object?
        isdict: bool = isinstance(tx, dict)
        # get best parser for this record..
        for k in parsers:
            if (
                (v := getattr(tx, k, None))
                or
                (
                    isdict
                    and
                    (v := tx.get(k))
                )
            ):
                # only call parser on the value if not None from
                # the `parsers` table above (when NOT using
                # `.get()`), otherwise pass through the value and
                # sort on it directly
                if (
                    not isinstance(v, DateTime)
                    and
                    (parser := parsers.get(k))
                ):
                    ret = parser(v)
                else:
                    ret = v
                return ret
            else:
                log.debug(
                    f'Parser-field not found in txn\n'
                    f'\n'
                    f'parser-field: {k!r}\n'
                    f'txn: {tx!r}\n'
                    f'\n'
                    f'Trying next..\n'
                )
                continue
        # XXX: we should never really get here bc it means some kinda
        # bad txn-record (field) data..
        #
        # -> set the `debug_mode = True` if you want to trace such
        # cases from REPL ;)
        else:
            # XXX: we should really never get here..
            # only if a ledger record has no expected sort(able)
            # field will we likely hit this.. like with ze IB.
            # if no sortable field just deliver epoch?
            log.warning(
                'No (time) sortable field for TXN:\n'
                f'{tx!r}\n'
            )
            report: str = (
                f'No supported time-field found in txn !?\n'
                f'\n'
                f'supported-time-fields: {parsers!r}\n'
                f'\n'
                f'txn: {tx!r}\n'
            )
            if debug:
                with maybe_open_crash_handler(
                    pdb=debug,
                    raise_on_exit=False,
                ):
                    raise ValueError(report)
            else:
                log.error(report)
            if _invalid is not None:
                _invalid.append(tx)
                return from_timestamp(0.)
    entry: tuple[str, dict]|Transaction
    invalid: list = []
    for entry in sorted(
        records,
        key=key or partial(
            dyn_parse_to_dt,
            _invalid=invalid,
        ),
    ):
        if entry in invalid:
            log.warning(
                f'Ignoring txn w invalid timestamp ??\n'
                f'{pformat(entry)}\n'
            )
            continue
        # NOTE the type sig above; either pairs or txns B)
        yield entry
 # TODO: probably just move this into the test suite or
 # keep it here for use from as such?
 # def ensure_state(self) -> None:
 #     '''
 #     Audit either the `.cumsize` and `.ppu` local instance vars against
 #     the clears table calculations and return the calc-ed values if
 #     they differ and log warnings to console.
 #     '''
 #     # clears: list[dict] = self._clears
 #     # self.first_clear_dt = min(clears, key=lambda e: e['dt'])['dt']
 #     last_clear: dict = clears[-1]
 #     csize: float = self.calc_size()
 #     accum: float = last_clear['accum_size']
 #     if not self.expired():
 #         if (
 #             csize != accum
 #             and csize != round(accum * (self.split_ratio or 1))
 #         ):
 #             raise ValueError(f'Size mismatch: {csize}')
 #     else:
 #         assert csize == 0, 'Contract is expired but non-zero size?'
 #     if self.cumsize != csize:
 #         log.warning(
 #             'Position state mismatch:\n'
 #             f'{self.cumsize} => {csize}'
 #         )
 #         self.cumsize = csize
 #     cppu: float = self.calc_ppu()
 #     ppu: float = last_clear['ppu']
 #     if (
 #         cppu != ppu
 #         and self.split_ratio is not None
 #         # handle any split info entered (for now) manually by user
 #         and cppu != (ppu / self.split_ratio)
 #     ):
 #         raise ValueError(f'PPU mismatch: {cppu}')
 #     if self.ppu != cppu:
 #         log.warning(
 #             'Position state mismatch:\n'
 #             f'{self.ppu} => {cppu}'
 #         )
 #         self.ppu = cppu
@cm
 def open_ledger_dfs(
    brokername: str,
    acctname: str,
    ledger: TransactionLedger | None = None,
    debug_mode: bool = False,
    **kwargs,
 ) -> tuple[
    dict[str, pl.DataFrame],
    TransactionLedger,
 ]:
    '''
    Open a ledger of trade records (presumably from some broker
    backend), normalize the records into `Transactions` via the
    backend's declared endpoint, cast to a `polars.DataFrame` which
    can update the ledger on exit.
    '''
    with maybe_open_crash_handler(
        pdb=debug_mode,
        # raise_on_exit=False,
    ):
        if not ledger:
            import time
            from ._ledger import open_trade_ledger
            now = time.time()
            with open_trade_ledger(
                    brokername,
                    acctname,
                    rewrite=True,
                    allow_from_sync_code=True,
                    # proxied through from caller
                    **kwargs,
            ) as ledger:
                if not ledger:
                    raise ValueError(f'No ledger for {acctname}@{brokername} exists?')
                print(f'LEDGER LOAD TIME: {time.time() - now}')
        yield ledger_to_dfs(ledger), ledger
 def ledger_to_dfs(
    ledger: TransactionLedger,
 ) -> dict[str, pl.DataFrame]:
    txns: dict[str, Transaction] = ledger.to_txns()
    # ldf = pl.DataFrame(
    #     list(txn.to_dict() for txn in txns.values()),
    ldf = pl.from_dicts(
        list(txn.to_dict() for txn in txns.values()),
        # only for ordering the cols
        schema=[
            ('fqme', str),
            ('tid', str),
            ('bs_mktid', str),
            ('expiry', str),
            ('etype', str),
            ('dt', str),
            ('size', pl.Float64),
            ('price', pl.Float64),
            ('cost', pl.Float64),
        ],
    ).sort(  # chronological order
        'dt'
    ).with_columns([
        pl.col('dt').str.to_datetime(),
        # pl.col('expiry').str.to_datetime(),
        # pl.col('expiry').dt.date(),
    ])
    # filter out to the columns matching values filter passed
    # as input.
    # if filter_by_ids:
    #     for col, vals in filter_by_ids.items():
    #         str_vals = set(map(str, vals))
    #         pred: pl.Expr = pl.col(col).eq(str_vals.pop())
    #         for val in str_vals:
    #             pred |= pl.col(col).eq(val)
    #     fdf = df.filter(pred)
    # TODO: originally i had tried just using a plain ol' groupby
    # + agg here but the issue was re-inserting to the src frame.
    # however, learning more about `polars` seems like maybe we can
    # use `.over()`?
    # https://pola-rs.github.io/polars/py-polars/html/reference/expressions/api/polars.Expr.over.html#polars.Expr.over
    # => CURRENTLY we break up into a frame per mkt / fqme
    dfs: dict[str, pl.DataFrame] = ldf.partition_by(
        'bs_mktid',
        as_dict=True,
    )
    # TODO: not sure if this is even possible but..
    # - it'd be more ideal to use `ppt = df.groupby('fqme').agg([`
    # - ppu and bep calcs!
    for key in dfs:
        # covert to lazy form (since apparently we might need it
        # eventually ...)
        df: pl.DataFrame = dfs[key]
        ldf: pl.LazyFrame = df.lazy()
        df = dfs[key] = ldf.with_columns([
            pl.cum_sum('size').alias('cumsize'),
            # amount of source asset "sent" (via buy txns in
            # the market) to acquire the dst asset, PER txn.
            # when this value is -ve (i.e. a sell operation) then
            # the amount sent is actually "returned".
            (
                (pl.col('price') * pl.col('size'))
                +
                (pl.col('cost')) # * pl.col('size').sign())
            ).alias('dst_bot'),
        ]).with_columns([
            # rolling balance in src asset units
            (pl.col('dst_bot').cum_sum() * -1).alias('src_balance'),
            # "position operation type" in terms of increasing the
            # amount in the dst asset (entering) or decreasing the
            # amount in the dst asset (exiting).
            pl.when(
                pl.col('size').sign() == pl.col('cumsize').sign()
            ).then(
                pl.lit('enter')  # see above, but is just price * size per txn
            ).otherwise(
                pl.when(pl.col('cumsize') == 0)
                .then(pl.lit('exit_to_zero'))
                .otherwise(pl.lit('exit'))
            ).alias('descr'),
            (pl.col('cumsize').sign() == pl.col('size').sign())
            .alias('is_enter'),
        ]).with_columns([
            # pl.lit(0, dtype=pl.Utf8).alias('virt_cost'),
            pl.lit(0, dtype=pl.Float64).alias('applied_cost'),
            pl.lit(0, dtype=pl.Float64).alias('pos_ppu'),
            pl.lit(0, dtype=pl.Float64).alias('per_txn_pnl'),
            pl.lit(0, dtype=pl.Float64).alias('cum_pos_pnl'),
            pl.lit(0, dtype=pl.Float64).alias('pos_bep'),
            pl.lit(0, dtype=pl.Float64).alias('cum_ledger_pnl'),
            pl.lit(None, dtype=pl.Float64).alias('ledger_bep'),
            # TODO: instead of the iterative loop below i guess we
            # could try using embedded lists to track which txns
            # are part of which ppu / bep calcs? Not sure this will
            # look any better nor be any more performant though xD
            # pl.lit([[0]], dtype=pl.List(pl.Float64)).alias('list'),
        # choose fields to emit for accounting puposes
        ]).select([
            pl.exclude([
                'tid',
                # 'dt',
                'expiry',
                'bs_mktid',
                'etype',
                # 'is_enter',
            ]),
        ]).collect()
        # compute recurrence relations for ppu and bep
        last_ppu: float = 0
        last_cumsize: float = 0
        last_ledger_pnl: float = 0
        last_pos_pnl: float = 0
        virt_costs: list[float, float] = [0., 0.]
        # imperatively compute the PPU (price per unit) and BEP
        # (break even price) iteratively over the ledger, oriented
        # around each position state: a state of split balances in
        # > 1 asset.
        for i, row in enumerate(df.iter_rows(named=True)):
            cumsize: float = row['cumsize']
            is_enter: bool = row['is_enter']
            price: float = row['price']
            size: float = row['size']
            # the profit is ALWAYS decreased, aka made a "loss"
            # by the constant fee charged by the txn provider!
            # see below in final PnL calculation and row element
            # set.
            txn_cost: float = row['cost']
            pnl: float = 0
            # ALWAYS reset per-position cum PnL
            if last_cumsize == 0:
                last_pos_pnl: float = 0
            # a "position size INCREASING" or ENTER transaction
            # which "makes larger", in src asset unit terms, the
            # trade's side-size of the destination asset:
            # - "buying" (more) units of the dst asset
            # - "selling" (more short) units of the dst asset
            if is_enter:
                # Naively include transaction cost in breakeven
                # price and presume the worst case of the
                # exact-same-cost-to-exit this transaction's worth
                # of size even though in reality it will be dynamic
                # based on exit strategy, price, liquidity, etc..
                virt_cost: float = txn_cost
                # cpu: float = cost / size
                # cummean of the cost-per-unit used for modelling
                # a projected future exit cost which we immediately
                # include in the costs incorporated to BEP on enters
                last_cum_costs_size, last_cpu = virt_costs
                cum_costs_size: float = last_cum_costs_size + abs(size)
                cumcpu = (
                    (last_cpu * last_cum_costs_size)
                    +
                    txn_cost
                ) / cum_costs_size
                virt_costs = [cum_costs_size, cumcpu]
                txn_cost = txn_cost + virt_cost
                # df[i, 'virt_cost'] = f'{-virt_cost} FROM {cumcpu}@{cum_costs_size}'
                # a cumulative mean of the price-per-unit acquired
                # in the destination asset:
                # https://en.wikipedia.org/wiki/Moving_average#Cumulative_average
                # You could also think of this measure more
                # generally as an exponential mean with `alpha
                # = 1/N` where `N` is the current number of txns
                # included in the "position" defining set:
                # https://en.wikipedia.org/wiki/Exponential_smoothing
                ppu: float = (
                    (
                        (last_ppu * last_cumsize)
                        +
                        (price * size)
                    ) /
                    cumsize
                )
            # a "position size DECREASING" or EXIT transaction
            # which "makes smaller" the trade's side-size of the
            # destination asset:
            # - selling previously bought units of the dst asset
            #   (aka 'closing' a long position).
            # - buying previously borrowed and sold (short) units
            #   of the dst asset (aka 'covering'/'closing' a short
            #   position).
            else:
                # only changes on position size increasing txns
                ppu: float = last_ppu
                # UNWIND IMPLIED COSTS FROM ENTRIES
                # => Reverse the virtual/modelled (2x predicted) txn
                # cost that was included in the least-recently
                # entered txn that is still part of the current CSi
                # set.
                # => we look up the cost-per-unit cum_sum and apply
                # if over the current txn size (by multiplication)
                # and then reverse that previusly applied cost on
                # the txn_cost for this record.
                #
                # NOTE: current "model" is just to previously assumed 2x
                # the txn cost for a matching enter-txn's
                # cost-per-unit; we then immediately reverse this
                # prediction and apply the real cost received here.
                last_cum_costs_size, last_cpu = virt_costs
                prev_virt_cost: float = last_cpu * abs(size)
                txn_cost: float = txn_cost - prev_virt_cost  # +ve thus a "reversal"
                cum_costs_size: float = last_cum_costs_size - abs(size)
                virt_costs = [cum_costs_size, last_cpu]
                # df[i, 'virt_cost'] = (
                #     f'{-prev_virt_cost} FROM {last_cpu}@{cum_costs_size}'
                # )
                # the per-txn profit or loss (PnL) given we are
                # (partially) "closing"/"exiting" the position via
                # this txn.
                pnl: float = (last_ppu - price) * size
            # always subtract txn cost from total txn pnl
            txn_pnl: float = pnl - txn_cost
            # cumulative PnLs per txn
            last_ledger_pnl = (
                last_ledger_pnl + txn_pnl
            )
            last_pos_pnl = df[i, 'cum_pos_pnl'] = (
                last_pos_pnl + txn_pnl
            )
            if cumsize == 0:
                last_ppu = ppu = 0
            # compute the BEP: "break even price", a value that
            # determines at what price the remaining cumsize can be
            # liquidated such that the net-PnL on the current
            # position will result in ZERO gain or loss from open
            # to close including all txn costs B)
            if (
                abs(cumsize) > 0  # non-exit-to-zero position txn
            ):
                cumsize_sign: float = copysign(1, cumsize)
                ledger_bep: float = (
                    (
                        (ppu * cumsize)
                        -
                        (last_ledger_pnl * cumsize_sign)
                    ) / cumsize
                )
                # NOTE: when we "enter more" dst asset units (aka
                # increase position state) AFTER having exited some
                # units (aka decreasing the pos size some) the bep
                # needs to be RECOMPUTED based on new ppu such that
                # liquidation of the cumsize at the bep price
                # results in a zero-pnl for the existing position
                # (since the last one).
                # for position lifetime BEP we never can have
                # a valid value once the position is "closed"
                # / full exitted Bo
                pos_bep: float = (
                    (
                        (ppu * cumsize)
                        -
                        (last_pos_pnl * cumsize_sign)
                    ) / cumsize
                )
            # inject DF row with all values
            df[i, 'pos_ppu'] = ppu
            df[i, 'per_txn_pnl'] = txn_pnl
            df[i, 'applied_cost'] = -txn_cost
            df[i, 'cum_pos_pnl'] = last_pos_pnl
            df[i, 'pos_bep'] = pos_bep
            df[i, 'cum_ledger_pnl'] = last_ledger_pnl
            df[i, 'ledger_bep'] = ledger_bep
            # keep backrefs to suffice reccurence relation
            last_ppu: float = ppu
            last_cumsize: float = cumsize
    # TODO?: pass back the current `Position` object loaded from
    # the account as well? Would provide incentive to do all
    # this ledger loading inside a new async open_account().
    # bs_mktid: str = df[0]['bs_mktid']
    # pos: Position = acnt.pps[bs_mktid]
    return dfs
--- a/piker/accounting/cli.py
+++ b/piker/accounting/cli.py
@ -1,318 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 CLI front end for trades ledger and position tracking management.
 '''
 from __future__ import annotations
 from pprint import pformat
 from rich.console import Console
 from rich.markdown import Markdown
 import polars as pl
 import tractor
 import trio
 import typer
 from piker.log import (
    get_console_log,
    get_logger,
 )
 from ..service import (
    open_piker_runtime,
 )
 from ..clearing._messages import BrokerdPosition
 from ..calc import humanize
 from ..brokers._daemon import broker_init
 from ._ledger import (
    load_ledger,
    TransactionLedger,
    # open_trade_ledger,
 )
 from .calc import (
    open_ledger_dfs,
 )
 log = get_logger(name=__name__)
 ledger = typer.Typer()
 def unpack_fqan(
    fully_qualified_account_name: str,
    console: Console | None = None,
 ) -> tuple | bool:
    try:
        brokername, account = fully_qualified_account_name.split('.')
        return brokername, account
    except ValueError:
        if console is not None:
            md = Markdown(
                f'=> `{fully_qualified_account_name}` <=\n\n'
                'is not a valid '
                '__fully qualified account name?__\n\n'
                'Your account name needs to be of the form '
                '`<brokername>.<account_name>`\n'
            )
            console.print(md)
        return False
@ledger.command()
 def sync(
    fully_qualified_account_name: str,
    pdb: bool = False,
    loglevel: str = typer.Option(
        'error',
        "-l",
    ),
 ):
    log = get_console_log(
        level=loglevel,
        name=__name__,
    )
    console = Console()
    pair: tuple[str, str]
    if not (pair := unpack_fqan(
        fully_qualified_account_name,
        console,
    )):
        return
    brokername, account = pair
    brokermod, start_kwargs, deamon_ep = broker_init(
        brokername,
        loglevel=loglevel,
    )
    brokername: str = brokermod.name
    async def main():
        async with (
            open_piker_runtime(
                name='ledger_cli',
                loglevel=loglevel,
                debug_mode=pdb,
            ) as (actor, sockaddr),
            tractor.open_nursery() as an,
        ):
            try:
                log.info(
                    f'Piker runtime up as {actor.uid}@{sockaddr}'
                )
                portal = await an.start_actor(
                    loglevel=loglevel,
                    debug_mode=pdb,
                    **start_kwargs,
                )
                from ..clearing import (
                    open_brokerd_dialog,
                )
                brokerd_stream: tractor.MsgStream
                async with (
                    # engage the brokerd daemon context
                    portal.open_context(
                        deamon_ep,
                        brokername=brokername,
                        loglevel=loglevel,
                    ),
                    # manually open the brokerd trade dialog EP
                    # (what the EMS normally does internall) B)
                    open_brokerd_dialog(
                        brokermod,
                        portal,
                        exec_mode=(
                            'paper'
                            if account == 'paper'
                            else 'live'
                        ),
                        loglevel=loglevel,
                    ) as (
                        brokerd_stream,
                        pp_msg_table,
                        accounts,
                    ),
                ):
                    try:
                        assert len(accounts) == 1
                        if not pp_msg_table:
                            ld, fpath = load_ledger(brokername, account)
                            assert not ld, f'WTF did we fail to parse ledger:\n{ld}'
                            console.print(
                                '[yellow]'
                                'No pps found for '
                                f'`{brokername}.{account}` '
                                'account!\n\n'
                                '[/][underline]'
                                'None of the following ledger files exist:\n\n[/]'
                                f'{fpath.as_uri()}\n'
                            )
                            return
                        pps_by_symbol: dict[str, BrokerdPosition] = pp_msg_table[
                            brokername,
                            account,
                        ]
                        summary: str = (
                            '[dim underline]Piker Position Summary[/] '
                            f'[dim blue underline]{brokername}[/]'
                            '[dim].[/]'
                            f'[blue underline]{account}[/]'
                            f'[dim underline] -> total pps: [/]'
                            f'[green]{len(pps_by_symbol)}[/]\n'
                        )
                        # for ppdict in positions:
                        for fqme, ppmsg in pps_by_symbol.items():
                            # ppmsg = BrokerdPosition(**ppdict)
                            size = ppmsg.size
                            if size:
                                ppu: float = round(
                                    ppmsg.avg_price,
                                    ndigits=2,
                                )
                                cost_basis: str = humanize(size * ppu)
                                h_size: str = humanize(size)
                                if size < 0:
                                    pcolor = 'red'
                                else:
                                    pcolor = 'green'
                                # sematic-highlight of fqme
                                fqme = ppmsg.symbol
                                tokens = fqme.split('.')
                                styled_fqme = f'[blue underline]{tokens[0]}[/]'
                                for tok in tokens[1:]:
                                    styled_fqme += '[dim].[/]'
                                    styled_fqme += f'[dim blue underline]{tok}[/]'
                                # TODO: instead display in a ``rich.Table``?
                                summary += (
                                    styled_fqme +
                                    '[dim]: [/]'
                                    f'[{pcolor}]{h_size}[/]'
                                    '[dim blue]u @[/]'
                                    f'[{pcolor}]{ppu}[/]'
                                    '[dim blue] = [/]'
                                    f'[{pcolor}]$ {cost_basis}\n[/]'
                                )
                        console.print(summary)
                    finally:
                        # exit via ctx cancellation.
                        brokerd_ctx: tractor.Context = brokerd_stream._ctx
                        await brokerd_ctx.cancel(timeout=1)
                    # TODO: once ported to newer tractor branch we should
                    # be able to do a loop like this:
                    # while brokerd_ctx.cancel_called_remote is None:
                    #     await trio.sleep(0.01)
                    #     await brokerd_ctx.cancel()
            finally:
                await portal.cancel_actor()
    trio.run(main)
@ledger.command()
 def disect(
    # "fully_qualified_account_name"
    fqan: str,
    fqme: str,  # for ib
    # TODO: in tractor we should really have
    # a debug_mode ctx for wrapping any kind of code no?
    pdb: bool = False,
    bs_mktid: str = typer.Option(
        None,
        "-bid",
    ),
    loglevel: str = typer.Option(
        'error',
        "-l",
    ),
 ):
    from piker.log import get_console_log
    from piker.toolz import open_crash_handler
    get_console_log(loglevel)
    pair: tuple[str, str]
    if not (pair := unpack_fqan(fqan)):
        raise ValueError('{fqan} malformed!?')
    brokername, account = pair
    # ledger dfs groupby-partitioned by fqme
    dfs: dict[str, pl.DataFrame]
    # actual ledger instance
    ldgr: TransactionLedger
    pl.Config.set_tbl_cols(-1)
    pl.Config.set_tbl_rows(-1)
    with (
        open_crash_handler(),
        open_ledger_dfs(
            brokername,
            account,
        ) as (dfs, ldgr),
    ):
        # look up specific frame for fqme-selected asset
        if (df := dfs.get(fqme)) is None:
            mktids2fqmes: dict[str, list[str]] = {}
            for bs_mktid in dfs:
                df: pl.DataFrame = dfs[bs_mktid]
                fqmes: pl.Series[str] = df['fqme']
                uniques: list[str] = fqmes.unique()
                mktids2fqmes[bs_mktid] = set(uniques)
                if fqme in uniques:
                    break
            print(
                f'No specific ledger for fqme={fqme} could be found in\n'
                f'{pformat(mktids2fqmes)}?\n'
                f'Maybe the `{brokername}` backend uses something '
                'else for its `bs_mktid` then the `fqme`?\n'
                'Scanning for matches in unique fqmes per frame..\n'
            )
        # :pray:
        assert not df.is_empty()
        # muck around in pdbp REPL
        # tractor.devx.mk_pdb().set_trace()
        # breakpoint()
        # TODO: we REALLY need a better console REPL for this
        # kinda thing..
        # - `xonsh` is an obvious option (and it looks amazin) but
        # we need to figure out how to embed it better then just:
        # from xonsh.main import main
        # main(argv=[])
        # which will not actually inject the `df` to globals?
--- a/piker/brokers/init.py
+++ b/piker/brokers/init.py
@ -17,40 +17,17 @@
 """
 Broker clients, daemons and general back end machinery.
 """
 from contextlib import (
    asynccontextmanager as acm,
 )
 from importlib import import_module
 from types import ModuleType
-from tractor.trionics import maybe_open_context
+# TODO: move to urllib3/requests once supported
 import asks
 asks.init('trio')
-from piker.log import (
+__brokers__ = [
    get_logger,
 )
 from ._util import (
    BrokerError,
    SymbolNotFound,
    NoData,
    DataUnavailable,
    DataThrottle,
    resproc,
 )
 __all__: list[str] = [
    'BrokerError',
    'SymbolNotFound',
    'NoData',
    'DataUnavailable',
    'DataThrottle',
    'resproc',
 ]
 __brokers__: list[str] = [
    'binance',
    'ib',
    'kraken',
    'kucoin',
    # broken but used to work
    # 'questrade',
@ -62,55 +39,22 @@ __brokers__: list[str] = [
    # iex
    # deribit
    # kucoin
    # bitso
 ]
 log = get_logger(
    name=__name__,
 )
 def get_brokermod(brokername: str) -> ModuleType:
-    '''
+    """Return the imported broker module by name.
-    Return the imported broker module by name.
+    """
-
+    module = import_module('.' + brokername, 'piker.brokers')
    '''
    module: ModuleType = import_module('.' + brokername, 'piker.brokers')
    # we only allow monkeying because it's for internal keying
-    module.name = module.__name__.split('.')[-1]
+    module.name =  module.__name__.split('.')[-1]
    return module
 def iter_brokermods():
-    '''
+    """Iterate all built-in broker modules.
-    Iterate all built-in broker modules.
+    """
    '''
    for name in __brokers__:
        yield get_brokermod(name)
@acm
 async def open_cached_client(
    brokername: str,
    **kwargs,
 ) -> 'Client':  # noqa
    '''
    Get a cached broker client from the current actor's local vars.
    If one has not been setup do it and cache it.
    '''
    brokermod: ModuleType = get_brokermod(brokername)
    # TODO: make abstract or `typing.Protocol`
    # client: Client
    async with maybe_open_context(
        acm_func=brokermod.get_client,
        kwargs=kwargs,
    ) as (cache_hit, client):
        if cache_hit:
            log.runtime(f'Reusing existing {client}')
        yield client
--- a/piker/brokers/_daemon.py
+++ b/piker/brokers/_daemon.py
@ -1,286 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Broker-daemon-actor "endpoint-hooks": the service task entry points for
 ``brokerd``.
 '''
 from __future__ import annotations
 from contextlib import (
    asynccontextmanager as acm,
 )
 from types import ModuleType
 from typing import (
    TYPE_CHECKING,
    AsyncContextManager,
 )
 import exceptiongroup as eg
 import tractor
 import trio
 from piker.log import (
    get_logger,
    get_console_log,
 )
 from . import _util
 from . import get_brokermod
 if TYPE_CHECKING:
    from ..data import _FeedsBus
 log = get_logger(name=__name__)
 # `brokerd` enabled modules
 # TODO: move this def to the `.data` subpkg..
 # NOTE: keeping this list as small as possible is part of our caps-sec
 # model and should be treated with utmost care!
 _data_mods: str = [
    'piker.brokers.core',
    'piker.brokers.data',
    'piker.brokers._daemon',
    'piker.data',
    'piker.data.feed',
    'piker.data._sampling'
 ]
 # TODO: we should rename the daemon to datad prolly once we split up
 # broker vs. data tasks into separate actors?
@tractor.context
 async def _setup_persistent_brokerd(
    ctx: tractor.Context,
    brokername: str,
    loglevel: str|None = None,
 ) -> None:
    '''
    Allocate a actor-wide service nursery in ``brokerd``
    such that feeds can be run in the background persistently by
    the broker backend as needed.
    '''
    # NOTE: we only need to setup logging once (and only) here
    # since all hosted daemon tasks will reference this same
    # log instance's (actor local) state and thus don't require
    # any further (level) configuration on their own B)
    actor: tractor.Actor = tractor.current_actor()
    tll: str = actor.loglevel
    log = get_console_log(
        level=loglevel or tll,
        name=f'{_util.subsys}.{brokername}',
        with_tractor_log=bool(tll),
    )
    assert log.name == _util.subsys
    # further, set the log level on any broker broker specific
    # logger instance.
    from piker.data import feed
    assert not feed._bus
    # allocate a nursery to the bus for spawning background
    # tasks to service client IPC requests, normally
    # `tractor.Context` connections to explicitly required
    # `brokerd` endpoints such as:
    # - `stream_quotes()`,
    # - `manage_history()`,
    # - `allocate_persistent_feed()`,
    # - `open_symbol_search()`
    # NOTE: see ep invocation details inside `.data.feed`.
    try:
        async with (
            # tractor.trionics.collapse_eg(),
            trio.open_nursery() as service_nursery
        ):
            bus: _FeedsBus = feed.get_feed_bus(
                brokername,
                service_nursery,
            )
            assert bus is feed._bus
            # unblock caller
            await ctx.started()
            # we pin this task to keep the feeds manager active until the
            # parent actor decides to tear it down
            await trio.sleep_forever()
    except eg.ExceptionGroup:
        # TODO: likely some underlying `brokerd` IPC connection
        # broke so here we handle a respawn and re-connect attempt!
        # This likely should pair with development of the OCO task
        # nusery in dev over @ `tractor` B)
        # https://github.com/goodboy/tractor/pull/363
        raise
 def broker_init(
    brokername: str,
    loglevel: str | None = None,
    **start_actor_kwargs,
 ) -> tuple[
    ModuleType,
    dict,
    AsyncContextManager,
 ]:
    '''
    Given an input broker name, load all named arguments
    which can be passed for daemon endpoint + context spawn
    as required in every `brokerd` (actor) service.
    This includes:
    - load the appropriate <brokername>.py pkg module,
    - reads any declared `__enable_modules__: listr[str]` which will be
      passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
      at actor start time,
    - deliver a references to the daemon lifetime fixture, which
      for now is always the `_setup_persistent_brokerd()` context defined
      above.
    '''
    from ..brokers import get_brokermod
    brokermod = get_brokermod(brokername)
    modpath: str = brokermod.__name__
    start_actor_kwargs['name'] = f'brokerd.{brokername}'
    start_actor_kwargs.update(
        getattr(
            brokermod,
            '_spawn_kwargs',
            {},
        )
    )
    # XXX TODO: make this not so hacky/monkeypatched..
    # -> we need a sane way to configure the logging level for all
    # code running in brokerd.
    # if utilmod := getattr(brokermod, '_util', False):
    #     utilmod.log.setLevel(loglevel.upper())
    # lookup actor-enabled modules declared by the backend offering the
    # `brokerd` endpoint(s).
    enabled: list[str]
    enabled = start_actor_kwargs['enable_modules'] = [
        __name__,  # so that eps from THIS mod can be invoked
        modpath,
    ]
    for submodname in getattr(
        brokermod,
        '__enable_modules__',
        [],
    ):
        subpath: str = f'{modpath}.{submodname}'
        enabled.append(subpath)
    return (
        brokermod,
        start_actor_kwargs,  # to `ActorNursery.start_actor()`
        # XXX see impl above; contains all (actor global)
        # setup/teardown expected in all `brokerd` actor instances.
        _setup_persistent_brokerd,
    )
 async def spawn_brokerd(
    brokername: str,
    loglevel: str | None = None,
    **tractor_kwargs,
 ) -> bool:
    log.info(
        f'Spawning broker-daemon,\n'
        f'backend: {brokername!r}'
    )
    (
        brokermode,
        tractor_kwargs,
        daemon_fixture_ep,
    ) = broker_init(
        brokername,
        loglevel,
        **tractor_kwargs,
    )
    brokermod = get_brokermod(brokername)
    extra_tractor_kwargs = getattr(brokermod, '_spawn_kwargs', {})
    tractor_kwargs.update(extra_tractor_kwargs)
    # ask `pikerd` to spawn a new sub-actor and manage it under its
    # actor nursery
    from piker.service import Services
    dname: str = tractor_kwargs.pop('name')  # f'brokerd.{brokername}'
    portal = await Services.actor_n.start_actor(
        dname,
        enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
        debug_mode=Services.debug_mode,
        **tractor_kwargs
    )
    # NOTE: the service mngr expects an already spawned actor + its
    # portal ref in order to do non-blocking setup of brokerd
    # service nursery.
    await Services.start_service_task(
        dname,
        portal,
        # signature of target root-task endpoint
        daemon_fixture_ep,
        brokername=brokername,
        loglevel=loglevel,
    )
    return True
@acm
 async def maybe_spawn_brokerd(
    brokername: str,
    loglevel: str|None = None,
    **pikerd_kwargs,
 ) -> tractor.Portal:
    '''
    Helper to spawn a brokerd service *from* a client who wishes to
    use the sub-actor-daemon but is fine with re-using any existing
    and contactable `brokerd`.
    Mas o menos, acts as a cached-actor-getter factory.
    '''
    from piker.service import maybe_spawn_daemon
    async with maybe_spawn_daemon(
        service_name=f'brokerd.{brokername}',
        service_task_target=spawn_brokerd,
        spawn_args={
            'brokername': brokername,
        },
        loglevel=loglevel,
        **pikerd_kwargs,
    ) as portal:
        yield portal
--- a/piker/brokers/_util.py
+++ b/piker/brokers/_util.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -15,40 +15,13 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-Handy cross-broker utils.
+Handy utils.
 """
 from __future__ import annotations
 # from functools import partial
 import json
-import httpx
+import asks
 import logging
-from piker.log import (
+from ..log import colorize_json
    colorize_json,
 )
 subsys: str = 'piker.brokers'
 # NOTE: level should be reset by any actor that is spawned
 # as well as given a (more) explicit name/key such
 # as `piker.brokers.binance` matching the subpkg.
 # log = get_logger(subsys)
 # ?TODO?? we could use this approach, but we need to be able
 # to pass multiple `name=` values so for example we can include the
 # emissions in `.accounting._pos` and others!
 # [ ] maybe we could do the `log = get_logger()` above,
 # then cycle through the list of subsys mods we depend on
 # and then get all their loggers and pass them to
 # `get_console_log(logger=)`??
 # [ ] OR just write THIS `get_console_log()` as a hook which does
 # that based on who calls it?.. i dunno
 #
 # get_console_log = partial(
 #     get_console_log,
 #     name=subsys,
 # )
 class BrokerError(Exception):
@ -59,7 +32,6 @@ class SymbolNotFound(BrokerError):
    "Symbol not found by broker search"
 # TODO: these should probably be moved to `.tsp/.data`?
 class NoData(BrokerError):
    '''
    Symbol data not permitted or no data
@ -69,15 +41,14 @@ class NoData(BrokerError):
    def __init__(
        self,
        *args,
-        info: dict|None = None,
+        frame_size: int = 1000,
    ) -> None:
        super().__init__(*args)
        self.info: dict|None = info
        # when raised, machinery can check if the backend
        # set a "frame size" for doing datetime calcs.
-        # self.frame_size: int = 1000
+        self.frame_size: int = 1000
 class DataUnavailable(BrokerError):
@ -98,19 +69,18 @@ class DataThrottle(BrokerError):
    # TODO: add in throttle metrics/feedback
 def resproc(
-    resp: httpx.Response,
+    resp: asks.response_objects.Response,
    log: logging.Logger,
    return_json: bool = True,
    log_resp: bool = False,
-) -> httpx.Response:
+) -> asks.response_objects.Response:
-    '''
+    """Process response and return its json content.
    Process response and return its json content.
    Raise the appropriate error on non-200 OK responses.
-
+    """
    '''
    if not resp.status_code == 200:
        raise BrokerError(resp.body)
    try:
--- a/piker/brokers/binance.py
+++ b/piker/brokers/binance.py
@ -0,0 +1,591 @@
 # piker: trading gear for hackers
 # Copyright (C) Guillermo Rodriguez (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 Binance backend
 """
 from contextlib import asynccontextmanager as acm
 from datetime import datetime
 from typing import (
    Any, Union, Optional,
    AsyncGenerator, Callable,
 )
 import time
 import trio
 from trio_typing import TaskStatus
 import pendulum
 import asks
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 import tractor
 import wsproto
 from .._cacheables import open_cached_client
 from ._util import (
    resproc,
    SymbolNotFound,
    DataUnavailable,
 )
 from ..log import (
    get_logger,
    get_console_log,
 )
 from ..data.types import Struct
 from ..data._web_bs import (
    open_autorecon_ws,
    NoBsWs,
 )
 log = get_logger(__name__)
 _url = 'https://api.binance.com'
 # Broker specific ohlc schema (rest)
 _ohlc_dtype = [
    ('index', int),
    ('time', int),
    ('open', float),
    ('high', float),
    ('low', float),
    ('close', float),
    ('volume', float),
    ('bar_wap', float),  # will be zeroed by sampler if not filled
    # XXX: some additional fields are defined in the docs:
    # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
    # ('close_time', int),
    # ('quote_vol', float),
    # ('num_trades', int),
    # ('buy_base_vol', float),
    # ('buy_quote_vol', float),
    # ('ignore', float),
 ]
 # UI components allow this to be declared such that additional
 # (historical) fields can be exposed.
 ohlc_dtype = np.dtype(_ohlc_dtype)
 _show_wap_in_history = False
 # https://binance-docs.github.io/apidocs/spot/en/#exchange-information
 class Pair(Struct, frozen=True):
    symbol: str
    status: str
    baseAsset: str
    baseAssetPrecision: int
    cancelReplaceAllowed: bool
    allowTrailingStop: bool
    quoteAsset: str
    quotePrecision: int
    quoteAssetPrecision: int
    baseCommissionPrecision: int
    quoteCommissionPrecision: int
    orderTypes: list[str]
    icebergAllowed: bool
    ocoAllowed: bool
    quoteOrderQtyMarketAllowed: bool
    isSpotTradingAllowed: bool
    isMarginTradingAllowed: bool
    defaultSelfTradePreventionMode: str
    allowedSelfTradePreventionModes: list[str]
    filters: list[dict[str, Union[str, int, float]]]
    permissions: list[str]
 class OHLC(Struct):
    '''
    Description of the flattened OHLC quote format.
    For schema details see:
    https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-streams
    '''
    time: int
    open: float
    high: float
    low: float
    close: float
    volume: float
    close_time: int
    quote_vol: float
    num_trades: int
    buy_base_vol: float
    buy_quote_vol: float
    ignore: int
    # null the place holder for `bar_wap` until we
    # figure out what to extract for this.
    bar_wap: float = 0.0
 # convert datetime obj timestamp to unixtime in milliseconds
 def binance_timestamp(
    when: datetime
 ) -> int:
    return int((when.timestamp() * 1000) + (when.microsecond / 1000))
 class Client:
    def __init__(self) -> None:
        self._sesh = asks.Session(connections=4)
        self._sesh.base_location = _url
        self._pairs: dict[str, Any] = {}
    async def _api(
        self,
        method: str,
        params: dict,
    ) -> dict[str, Any]:
        resp = await self._sesh.get(
            path=f'/api/v3/{method}',
            params=params,
            timeout=float('inf')
        )
        return resproc(resp, log)
    async def symbol_info(
        self,
        sym: Optional[str] = None,
    ) -> dict[str, Any]:
        '''Get symbol info for the exchange.
        '''
        # TODO: we can load from our self._pairs cache
        # on repeat calls...
        # will retrieve all symbols by default
        params = {}
        if sym is not None:
            sym = sym.lower()
            params = {'symbol': sym}
        resp = await self._api(
            'exchangeInfo',
            params=params,
        )
        entries = resp['symbols']
        if not entries:
            raise SymbolNotFound(f'{sym} not found')
        syms = {item['symbol']: item for item in entries}
        if sym is not None:
            return syms[sym]
        else:
            return syms
    async def cache_symbols(
        self,
    ) -> dict:
        if not self._pairs:
            self._pairs = await self.symbol_info()
        return self._pairs
    async def search_symbols(
        self,
        pattern: str,
        limit: int = None,
    ) -> dict[str, Any]:
        if self._pairs is not None:
            data = self._pairs
        else:
            data = await self.symbol_info()
        matches = fuzzy.extractBests(
            pattern,
            data,
            score_cutoff=50,
        )
        # repack in dict form
        return {item[0]['symbol']: item[0]
                for item in matches}
    async def bars(
        self,
        symbol: str,
        start_dt: Optional[datetime] = None,
        end_dt: Optional[datetime] = None,
        limit: int = 1000,  # <- max allowed per query
        as_np: bool = True,
    ) -> dict:
        if end_dt is None:
            end_dt = pendulum.now('UTC').add(minutes=1)
        if start_dt is None:
            start_dt = end_dt.start_of(
                'minute').subtract(minutes=limit)
        start_time = binance_timestamp(start_dt)
        end_time = binance_timestamp(end_dt)
        # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
        bars = await self._api(
            'klines',
            params={
                'symbol': symbol.upper(),
                'interval': '1m',
                'startTime': start_time,
                'endTime': end_time,
                'limit': limit
            }
        )
        # TODO: pack this bars scheme into a ``pydantic`` validator type:
        # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
        # TODO: we should port this to ``pydantic`` to avoid doing
        # manual validation ourselves..
        new_bars = []
        for i, bar in enumerate(bars):
            bar = OHLC(*bar)
            bar.typecast()
            row = []
            for j, (name, ftype) in enumerate(_ohlc_dtype[1:]):
                # TODO: maybe we should go nanoseconds on all
                # history time stamps?
                if name == 'time':
                    # convert to epoch seconds: float
                    row.append(bar.time / 1000.0)
                else:
                    row.append(getattr(bar, name))
            new_bars.append((i,) + tuple(row))
        array = np.array(new_bars, dtype=_ohlc_dtype) if as_np else bars
        return array
@acm
 async def get_client() -> Client:
    client = Client()
    await client.cache_symbols()
    yield client
 # validation type
 class AggTrade(Struct):
    e: str  # Event type
    E: int  # Event time
    s: str  # Symbol
    a: int  # Aggregate trade ID
    p: float  # Price
    q: float  # Quantity
    f: int  # First trade ID
    l: int  # Last trade ID
    T: int  # Trade time
    m: bool  # Is the buyer the market maker?
    M: bool  # Ignore
 async def stream_messages(ws: NoBsWs) -> AsyncGenerator[NoBsWs, dict]:
    timeouts = 0
    while True:
        with trio.move_on_after(3) as cs:
            msg = await ws.recv_msg()
        if cs.cancelled_caught:
            timeouts += 1
            if timeouts > 2:
                log.error("binance feed seems down and slow af? rebooting...")
                await ws._connect()
            continue
        # for l1 streams binance doesn't add an event type field so
        # identify those messages by matching keys
        # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
        if msg.get('u'):
            sym = msg['s']
            bid = float(msg['b'])
            bsize = float(msg['B'])
            ask = float(msg['a'])
            asize = float(msg['A'])
            yield 'l1', {
                'symbol': sym,
                'ticks': [
                    {'type': 'bid', 'price': bid, 'size': bsize},
                    {'type': 'bsize', 'price': bid, 'size': bsize},
                    {'type': 'ask', 'price': ask, 'size': asize},
                    {'type': 'asize', 'price': ask, 'size': asize}
                ]
            }
        elif msg.get('e') == 'aggTrade':
            # NOTE: this is purely for a definition, ``msgspec.Struct``
            # does not runtime-validate until you decode/encode.
            # see: https://jcristharif.com/msgspec/structs.html#type-validation
            msg = AggTrade(**msg)
            # TODO: type out and require this quote format
            # from all backends!
            yield 'trade', {
                'symbol': msg.s,
                'last': msg.p,
                'brokerd_ts': time.time(),
                'ticks': [{
                    'type': 'trade',
                    'price': float(msg.p),
                    'size': float(msg.q),
                    'broker_ts': msg.T,
                }],
            }
 def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
    """Create a request subscription packet dict.
    https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
    """
    return {
        'method': 'SUBSCRIBE',
        'params': [
            f'{pair.lower()}@{sub_name}'
            for pair in pairs
        ],
        'id': uid
    }
@acm
 async def open_history_client(
    symbol: str,
 ) -> tuple[Callable, int]:
    # TODO implement history getter for the new storage layer.
    async with open_cached_client('binance') as client:
        async def get_ohlc(
            timeframe: float,
            end_dt: datetime | None = None,
            start_dt: datetime | None = None,
        ) -> tuple[
            np.ndarray,
            datetime,  # start
            datetime,  # end
        ]:
            if timeframe != 60:
                raise DataUnavailable('Only 1m bars are supported')
            array = await client.bars(
                symbol,
                start_dt=start_dt,
                end_dt=end_dt,
            )
            times = array['time']
            if (
                end_dt is None
            ):
                inow = round(time.time())
                if (inow - times[-1]) > 60:
                    await tractor.breakpoint()
            start_dt = pendulum.from_timestamp(times[0])
            end_dt = pendulum.from_timestamp(times[-1])
            return array, start_dt, end_dt
        yield get_ohlc, {'erlangs': 3, 'rate': 3}
 async def stream_quotes(
    send_chan: trio.abc.SendChannel,
    symbols: list[str],
    feed_is_live: trio.Event,
    loglevel: str = None,
    # startup sync
    task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
 ) -> None:
    # XXX: required to propagate ``tractor`` loglevel to piker logging
    get_console_log(loglevel or tractor.current_actor().loglevel)
    sym_infos = {}
    uid = 0
    async with (
        open_cached_client('binance') as client,
        send_chan as send_chan,
    ):
        # keep client cached for real-time section
        cache = await client.cache_symbols()
        for sym in symbols:
            d = cache[sym.upper()]
            syminfo = Pair(**d)  # validation
            si = sym_infos[sym] = syminfo.to_dict()
            filters = {}
            for entry in syminfo.filters:
                ftype = entry['filterType']
                filters[ftype] = entry
            # XXX: after manually inspecting the response format we
            # just directly pick out the info we need
            si['price_tick_size'] = float(
                filters['PRICE_FILTER']['tickSize']
            )
            si['lot_tick_size'] = float(
                filters['LOT_SIZE']['stepSize']
            )
            si['asset_type'] = 'crypto'
        symbol = symbols[0]
        init_msgs = {
            # pass back token, and bool, signalling if we're the writer
            # and that history has been written
            symbol: {
                'symbol_info': sym_infos[sym],
                'shm_write_opts': {'sum_tick_vml': False},
                'fqsn': sym,
            },
        }
        @acm
        async def subscribe(ws: wsproto.WSConnection):
            # setup subs
            # trade data (aka L1)
            # https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
            l1_sub = make_sub(symbols, 'bookTicker', uid)
            await ws.send_msg(l1_sub)
            # aggregate (each order clear by taker **not** by maker)
            # trades data:
            # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
            agg_trades_sub = make_sub(symbols, 'aggTrade', uid)
            await ws.send_msg(agg_trades_sub)
            # ack from ws server
            res = await ws.recv_msg()
            assert res['id'] == uid
            yield
            subs = []
            for sym in symbols:
                subs.append("{sym}@aggTrade")
                subs.append("{sym}@bookTicker")
            # unsub from all pairs on teardown
            if ws.connected():
                await ws.send_msg({
                    "method": "UNSUBSCRIBE",
                    "params": subs,
                    "id": uid,
                })
                # XXX: do we need to ack the unsub?
                # await ws.recv_msg()
        async with open_autorecon_ws(
            'wss://stream.binance.com/ws',
            fixture=subscribe,
        ) as ws:
            # pull a first quote and deliver
            msg_gen = stream_messages(ws)
            typ, quote = await msg_gen.__anext__()
            while typ != 'trade':
                # TODO: use ``anext()`` when it lands in 3.10!
                typ, quote = await msg_gen.__anext__()
            task_status.started((init_msgs,  quote))
            # signal to caller feed is ready for consumption
            feed_is_live.set()
            # import time
            # last = time.time()
            # start streaming
            async for typ, msg in msg_gen:
                # period = time.time() - last
                # hz = 1/period if period else float('inf')
                # if hz > 60:
                #     log.info(f'Binance quotez : {hz}')
                topic = msg['symbol'].lower()
                await send_chan.send({topic: msg})
                # last = time.time()
@tractor.context
 async def open_symbol_search(
    ctx: tractor.Context,
 ) -> Client:
    async with open_cached_client('binance') as client:
        # load all symbols locally for fast search
        cache = await client.cache_symbols()
        await ctx.started()
        async with ctx.open_stream() as stream:
            async for pattern in stream:
                # results = await client.symbol_info(sym=pattern.upper())
                matches = fuzzy.extractBests(
                    pattern,
                    cache,
                    score_cutoff=50,
                )
                # repack in dict form
                await stream.send(
                    {item[0]['symbol']: item[0]
                     for item in matches}
                )
--- a/piker/brokers/binance/init.py
+++ b/piker/brokers/binance/init.py
@ -1,60 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C)
 #   Guillermo Rodriguez (aka ze jefe)
 #   Tyler Goodlet
 #   (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 binancial secs on the floor, in the office, behind the dumpster.
 """
 from .api import (
    get_client,
 )
 from .feed import (
    get_mkt_info,
    open_history_client,
    open_symbol_search,
    stream_quotes,
 )
 from .broker import (
    open_trade_dialog,
    get_cost,
 )
 from .venues import (
    SpotPair,
    FutesPair,
 )
 __all__ = [
    'get_client',
    'get_mkt_info',
    'get_cost',
    'SpotPair',
    'FutesPair',
    'open_trade_dialog',
    'open_history_client',
    'open_symbol_search',
    'stream_quotes',
 ]
 # `brokerd` modules
 __enable_modules__: list[str] = [
    'api',
    'feed',
    'broker',
 ]
--- a/piker/brokers/binance/api.py
+++ b/piker/brokers/binance/api.py
--- a/piker/brokers/binance/broker.py
+++ b/piker/brokers/binance/broker.py
@ -1,721 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C)
 #   Guillermo Rodriguez (aka ze jefe)
 #   Tyler Goodlet
 #   (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Live order control B)
 '''
 from __future__ import annotations
 from pprint import pformat
 from typing import (
    Any,
    AsyncIterator,
 )
 import time
 from time import time_ns
 from bidict import bidict
 import tractor
 import trio
 from piker.accounting import (
    Asset,
 )
 from piker.log import (
    get_logger,
    get_console_log,
 )
 from piker.data._web_bs import (
    open_autorecon_ws,
    NoBsWs,
 )
 from piker.brokers import (
    open_cached_client,
    BrokerError,
 )
 from piker.clearing import (
    OrderDialogs,
 )
 from piker.clearing._messages import (
    BrokerdOrder,
    BrokerdOrderAck,
    BrokerdStatus,
    BrokerdPosition,
    BrokerdFill,
    BrokerdCancel,
    BrokerdError,
    Status,
    Order,
 )
 from .venues import (
    Pair,
    _futes_ws,
    _testnet_futes_ws,
 )
 from .api import Client
 log = get_logger(
    name=__name__,
 )
 # Fee schedule template, mostly for paper engine fees modelling.
 # https://www.binance.com/en/support/faq/what-are-market-makers-and-takers-360007720071
 def get_cost(
    price: float,
    size: float,
    is_taker: bool = False,
 ) -> float:
    # https://www.binance.com/en/fee/trading
    cb: float = price * size
    match is_taker:
        case True:
            return cb * 0.001000
        case False if cb < 1e6:
            return cb * 0.001000
        case False if 1e6 >= cb < 5e6:
            return cb * 0.000900
        # NOTE: there's more but are you really going
        # to have a cb bigger then this per trade?
        case False if cb >= 5e6:
            return cb * 0.000800
 async def handle_order_requests(
    ems_order_stream: tractor.MsgStream,
    client: Client,
    dids: bidict[str, str],
    dialogs: OrderDialogs,
 ) -> None:
    '''
    Receive order requests from `emsd`, translate tramsit API calls and transmit.
    '''
    msg: dict | BrokerdOrder | BrokerdCancel
    async for msg in ems_order_stream:
        log.info(f'Rx order request:\n{pformat(msg)}')
        match msg:
            case {
                'action': 'cancel',
            }:
                cancel = BrokerdCancel(**msg)
                existing: BrokerdOrder | None = dialogs.get(cancel.oid)
                if not existing:
                    log.error(
                        f'NO Existing order-dialog for {cancel.oid}!?'
                    )
                    await ems_order_stream.send(BrokerdError(
                        oid=cancel.oid,
                        # TODO: do we need the symbol?
                        # https://github.com/pikers/piker/issues/514
                        symbol='unknown',
                        reason=(
                            'Invalid `binance` order request dialog oid',
                        )
                    ))
                    continue
                else:
                    symbol: str = existing['symbol']
                    try:
                        await client.submit_cancel(
                            symbol,
                            cancel.oid,
                        )
                    except BrokerError as be:
                        await ems_order_stream.send(
                            BrokerdError(
                                oid=msg['oid'],
                                symbol=symbol,
                                reason=(
                                    '`binance` CANCEL failed:\n'
                                    f'{be}'
                                ))
                        )
                        continue
            case {
                'account': ('binance.usdtm' | 'binance.spot') as account,
                'action': action,
            } if action in {'buy', 'sell'}:
                # validate
                order = BrokerdOrder(**msg)
                oid: str = order.oid  # emsd order id
                modify: bool = False
                # NOTE: check and report edits
                if existing := dialogs.get(order.oid):
                    log.info(
                        f'Existing order for {oid} updated:\n'
                        f'{pformat(existing.maps[-1])} -> {pformat(msg)}'
                    )
                    modify = True
                    # only add new msg AFTER the existing check
                    dialogs.add_msg(oid, msg)
                else:
                    # XXX NOTE: update before the ack!
                    # track latest request state such that map
                    # lookups start at the most recent msg and then
                    # scan reverse-chronologically.
                    dialogs.add_msg(oid, msg)
                    # XXX: ACK the request **immediately** before sending
                    # the api side request to ensure the ems maps the oid ->
                    # reqid correctly!
                    resp = BrokerdOrderAck(
                        oid=oid,  # ems order request id
                        reqid=oid,  # our custom int mapping
                        account='binance',  # piker account
                    )
                    await ems_order_stream.send(resp)
                # call our client api to submit the order
                # NOTE: modifies only require diff key for user oid:
                # https://binance-docs.github.io/apidocs/futures/en/#modify-order-trade
                try:
                    reqid = await client.submit_limit(
                        symbol=order.symbol,
                        side=order.action,
                        quantity=order.size,
                        price=order.price,
                        oid=oid,
                        modify=modify,
                    )
                    # SMH they do gen their own order id: ints.. 
                    # assert reqid == order.oid
                    dids[order.oid] = reqid
                except BrokerError as be:
                    await ems_order_stream.send(
                        BrokerdError(
                            oid=msg['oid'],
                            symbol=msg['symbol'],
                            reason=(
                                '`binance` request failed:\n'
                                f'{be}'
                            ))
                    )
                    continue
            case _:
                account = msg.get('account')
                if account not in {'binance.spot', 'binance.futes'}:
                    log.error(
                        'Order request does not have a valid binance account name?\n'
                        'Only one of\n'
                        '- `binance.spot` or,\n'
                        '- `binance.usdtm`\n'
                        'is currently valid!'
                    )
                await ems_order_stream.send(
                    BrokerdError(
                        oid=msg['oid'],
                        symbol=msg['symbol'],
                        reason=(
                            f'Invalid `binance` broker request msg:\n{msg}'
                        ))
                )
@tractor.context
 async def open_trade_dialog(
    ctx: tractor.Context,
    loglevel: str = 'warning',
 ) -> AsyncIterator[dict[str, Any]]:
    # enable piker.clearing console log for *this* `brokerd` subactor
    get_console_log(
        level=loglevel,
        name=__name__,
    )
    # TODO: how do we set this from the EMS such that
    # positions are loaded from the correct venue on the user
    # stream at startup? (that is in an attempt to support both
    # spot and futes markets?)
    # - I guess we just want to instead start 2 separate user
    #   stream tasks right? unless we want another actor pool?
    #   XXX: see issue: <urlhere>
    venue_name: str = 'futes'
    venue_mode: str = 'usdtm_futes'
    account_name: str = 'usdtm'
    use_testnet: bool = False
    # TODO: if/when we add .accounting support we need to
    # do a open_symcache() call.. though maybe we can hide
    # this in a new async version of open_account()?
    async with open_cached_client('binance') as client:
        subconf: dict|None = client.conf.get(venue_name)
        # XXX: if no futes.api_key or spot.api_key has been set we
        # always fall back to the paper engine!
        if (
            not subconf
            or
            not subconf.get('api_key')
        ):
            await ctx.started('paper')
            return
        use_testnet: bool = subconf.get('use_testnet', False)
    async with (
        open_cached_client('binance') as client,
    ):
        client.mkt_mode: str = venue_mode
        # TODO: map these wss urls depending on spot or futes
        # setting passed when this task is spawned?
        wss_url: str = _futes_ws if not use_testnet else _testnet_futes_ws
        wss: NoBsWs
        async with (
            client.manage_listen_key() as listen_key,
            open_autorecon_ws(f'{wss_url}/?listenKey={listen_key}') as wss,
        ):
            nsid: int = time_ns()
            await wss.send_msg({
                # "method": "SUBSCRIBE",
                "method": "REQUEST",
                "params":
                [
                    f"{listen_key}@account",
                    f"{listen_key}@balance",
                    f"{listen_key}@position",
                    # TODO: does this even work!? seems to cause
                    # a hang on the first msg..? lelelel.
                    # f"{listen_key}@order",
                ],
                "id": nsid
            })
            with trio.fail_after(6):
                msg = await wss.recv_msg()
                assert msg['id'] == nsid
            # TODO: load other market wide data / statistics:
            # - OI: https://binance-docs.github.io/apidocs/futures/en/#open-interest
            # - OI stats: https://binance-docs.github.io/apidocs/futures/en/#open-interest-statistics
            accounts: bidict[str, str] = bidict({'binance.usdtm': None})
            balances: dict[Asset, float] = {}
            positions: list[BrokerdPosition] = []
            for resp_dict in msg['result']:
                resp: dict = resp_dict['res']
                req: str = resp_dict['req']
                # @account response should be something like:
                # {'accountAlias': 'sRFzFzAuuXsR',
                #  'canDeposit': True,
                #  'canTrade': True,
                #  'canWithdraw': True,
                #  'feeTier': 0}
                if 'account' in req:
                    # NOTE: fill in the hash-like key/alias binance
                    # provides for the account.
                    alias: str = resp['accountAlias']
                    accounts['binance.usdtm'] = alias
                # @balance response:
                # {'accountAlias': 'sRFzFzAuuXsR',
                #      'balances': [{'asset': 'BTC',
                #                    'availableBalance': '0.00000000',
                #                    'balance': '0.00000000',
                #                    'crossUnPnl': '0.00000000',
                #                    'crossWalletBalance': '0.00000000',
                #                    'maxWithdrawAmount': '0.00000000',
                #                    'updateTime': 0}]
                #                     ...
                # }
                elif 'balance' in req:
                    for entry in resp['balances']:
                        name: str = entry['asset']
                        balance: float = float(entry['balance'])
                        last_update_t: int = entry['updateTime']
                        spot_asset: Asset = client._venue2assets['spot'][name]
                        if balance > 0:
                            balances[spot_asset] = (balance, last_update_t)
                            # await tractor.pause()
                # @position response:
                # {'positions': [{'entryPrice': '0.0',
                #                    'isAutoAddMargin': False,
                #                    'isolatedMargin': '0',
                #                    'leverage': 20,
                #                    'liquidationPrice': '0',
                #                    'marginType': 'CROSSED',
                #                    'markPrice': '0.60289650',
                #                    'markPrice': '0.00000000',
                #                    'maxNotionalValue': '25000',
                #                    'notional': '0',
                #                    'positionAmt': '0',
                #                    'positionSide': 'BOTH',
                #                    'symbol': 'ETHUSDT_230630',
                #                    'unRealizedProfit': '0.00000000',
                #                    'updateTime': 1672741444894}
                #                    ...
                # }
                elif 'position' in req:
                    for entry in resp['positions']:
                        bs_mktid: str = entry['symbol']
                        entry_size: float = float(entry['positionAmt'])
                        pair: Pair | None = client._venue2pairs[
                            venue_mode
                        ].get(bs_mktid)
                        if (
                            pair
                            and entry_size > 0
                        ):
                            entry_price: float = float(entry['entryPrice'])
                            ppmsg = BrokerdPosition(
                                broker='binance',
                                account=f'binance.{account_name}',
                                # TODO: maybe we should be passing back
                                # a `MktPair` here?
                                symbol=pair.bs_fqme.lower() + '.binance',
                                size=entry_size,
                                avg_price=entry_price,
                            )
                            positions.append(ppmsg)
                        if pair is None:
                            log.warning(
                                f'`{bs_mktid}` Position entry but no market pair?\n'
                                f'{pformat(entry)}\n'
                            )
            await ctx.started((
                positions,
                list(accounts)
            ))
            # TODO: package more state tracking into the dialogs API?
            # - hmm maybe we could include `OrderDialogs.dids:
            #   bidict` as part of the interface and then ask for
            #   a reqid field to be passed at init?
            #   |-> `OrderDialog(reqid_field='orderId')` kinda thing?
            # - also maybe bundle in some kind of dialog to account
            #   table?
            dialogs = OrderDialogs()
            dids: dict[str, int] = bidict()
            # TODO: further init setup things to get full EMS and
            # .accounting support B)
            # - live order loading via user stream subscription and
            #   update to the order dialog table.
            #   - MAKE SURE we add live orders loaded during init
            #   into the dialogs table to ensure they can be
            #   cancelled, meaning we can do a symbol lookup.
            # - position loading using `piker.accounting` subsys
            #   and comparison with binance's own position calcs.
            # - load pps and accounts using accounting apis, write
            #   the ledger and account files
            #   - table: Account
            #   - ledger: TransactionLedger
            async with (
                tractor.trionics.collapse_eg(),
                trio.open_nursery() as tn,
                ctx.open_stream() as ems_stream,
            ):
                # deliver all pre-exist open orders to EMS thus syncing
                # state with existing live limits reported by them.
                order: Order
                for order in await client.get_open_orders():
                    status_msg = Status(
                        time_ns=time.time_ns(),
                        resp='open',
                        oid=order.oid,
                        reqid=order.oid,
                        # embedded order info
                        req=order,
                        src='binance',
                    )
                    dialogs.add_msg(order.oid, order.to_dict())
                    await ems_stream.send(status_msg)
                tn.start_soon(
                    handle_order_requests,
                    ems_stream,
                    client,
                    dids,
                    dialogs,
                )
                tn.start_soon(
                    handle_order_updates,
                    venue_mode,
                    account_name,
                    client,
                    ems_stream,
                    wss,
                    dialogs,
                )
                await trio.sleep_forever()
 async def handle_order_updates(
    venue: str,
    account_name: str,
    client: Client,
    ems_stream: tractor.MsgStream,
    wss: NoBsWs,
    dialogs: OrderDialogs,
 ) -> None:
    '''
    Main msg handling loop for all things order management.
    This code is broken out to make the context explicit and state
    variables defined in the signature clear to the reader.
    '''
    async for msg in wss:
        log.info(f'Rx USERSTREAM msg:\n{pformat(msg)}')
        match msg:
            # ORDER update
            # spot: https://binance-docs.github.io/apidocs/spot/en/#payload-balance-update
            # futes: https://binance-docs.github.io/apidocs/futures/en/#event-order-update
            # futes: https://binance-docs.github.io/apidocs/futures/en/#event-balance-and-position-update
            # {'o': {
            #    'L': '0',
            #    'N': 'USDT',
            #    'R': False,
            #    'S': 'BUY',
            #    'T': 1687028772484,
            #    'X': 'NEW',
            #    'a': '0',
            #    'ap': '0',
            #    'b': '7012.06520',
            #    'c': '518d4122-8d3e-49b0-9a1e-1fabe6f62e4c',
            #    'cp': False,
            #    'f': 'GTC',
            #    'i': 3376956924,
            #    'l': '0',
            #    'm': False,
            #    'n': '0',
            #    'o': 'LIMIT',
            #    'ot': 'LIMIT',
            #    'p': '21136.80',
            #    'pP': False,
            #    'ps': 'BOTH',
            #    'q': '0.047',
            #    'rp': '0',
            #    's': 'BTCUSDT',
            #    'si': 0,
            #    'sp': '0',
            #    'ss': 0,
            #    't': 0,
            #    'wt': 'CONTRACT_PRICE',
            #    'x': 'NEW',
            #    'z': '0'}
            # }
            case {
                # 'e': 'executionReport',
                'e': 'ORDER_TRADE_UPDATE',
                'T': int(epoch_ms),
                'o': {
                    's': bs_mktid,
                    # XXX NOTE XXX see special ids for market
                    # events or margin calls:
                    # // special client order id:
                    # // starts with "autoclose-": liquidation order
                    # // "adl_autoclose": ADL auto close order
                    # // "settlement_autoclose-": settlement order
                    #     for delisting or delivery
                    'c': oid,
                    # 'i': reqid,  # binance internal int id
                    # prices
                    'a': submit_price,
                    'ap': avg_price,
                    'L': fill_price,
                    # sizing
                    'q': req_size,
                    'l': clear_size_filled,  # this event
                    'z': accum_size_filled,  # accum
                    # commissions
                    'n': cost,
                    'N': cost_asset,
                    # state
                    'S': side,
                    'X': status,
                },
            } as order_msg:
                log.info(
                    f'{status} for {side} ORDER oid: {oid}\n'
                    f'bs_mktid: {bs_mktid}\n\n'
                    f'order size: {req_size}\n'
                    f'cleared size: {clear_size_filled}\n'
                    f'accum filled size: {accum_size_filled}\n\n'
                    f'submit price: {submit_price}\n'
                    f'fill_price: {fill_price}\n'
                    f'avg clearing price: {avg_price}\n\n'
                    f'cost: {cost}@{cost_asset}\n'
                )
                # status remap from binance to piker's
                # status set:
                # - NEW
                # - PARTIALLY_FILLED
                # - FILLED
                # - CANCELED
                # - EXPIRED
                # https://binance-docs.github.io/apidocs/futures/en/#event-order-update
                req_size: float = float(req_size)
                accum_size_filled: float = float(accum_size_filled)
                fill_price: float = float(fill_price)
                match status:
                    case 'PARTIALLY_FILLED' | 'FILLED':
                        status = 'fill'
                        fill_msg = BrokerdFill(
                            time_ns=time_ns(),
                            # reqid=reqid,
                            reqid=oid,
                            # just use size value for now?
                            # action=action,
                            size=clear_size_filled,
                            price=fill_price,
                            # TODO: maybe capture more msg data
                            # i.e fees?
                            broker_details={'name': 'broker'} | order_msg,
                            broker_time=time.time(),
                        )
                        await ems_stream.send(fill_msg)
                        if accum_size_filled == req_size:
                            status = 'closed'
                            dialogs.pop(oid)
                    case 'NEW':
                        status = 'open'
                    case 'EXPIRED':
                        status = 'canceled'
                        dialogs.pop(oid)
                    case _:
                        status = status.lower()
                resp = BrokerdStatus(
                    time_ns=time_ns(),
                    # reqid=reqid,
                    reqid=oid,
                    # TODO: i feel like we don't need to make the
                    # ems and upstream clients aware of this?
                    # account='binance.usdtm',
                    status=status,
                    filled=accum_size_filled,
                    remaining=req_size - accum_size_filled,
                    broker_details={
                        'name': 'binance',
                        'broker_time': epoch_ms / 1000.
                    }
                )
                await ems_stream.send(resp)
            # ACCOUNT and POSITION update B)
            # {
            #  'E': 1687036749218,
            #  'e': 'ACCOUNT_UPDATE'
            #  'T': 1687036749215,
            #  'a': {'B': [{'a': 'USDT',
            #               'bc': '0',
            #               'cw': '1267.48920735',
            #               'wb': '1410.90245576'}],
            #        'P': [{'cr': '-3292.10973007',
            #               'ep': '26349.90000',
            #               'iw': '143.41324841',
            #               'ma': 'USDT',
            #               'mt': 'isolated',
            #               'pa': '0.038',
            #               'ps': 'BOTH',
            #               's': 'BTCUSDT',
            #               'up': '5.17555453'}],
            #        'm': 'ORDER'},
            # }
            case {
                'T': int(epoch_ms),
                'e': 'ACCOUNT_UPDATE',
                'a': {
                    'P': [{
                        's': bs_mktid,
                        'pa': pos_amount,
                        'ep': entry_price,
                    }],
                },
            }:
                # real-time relay position updates back to EMS
                pair: Pair | None = client._venue2pairs[venue].get(bs_mktid)
                ppmsg = BrokerdPosition(
                    broker='binance',
                    account=f'binance.{account_name}',
                    # TODO: maybe we should be passing back
                    # a `MktPair` here?
                    symbol=pair.bs_fqme.lower() + '.binance',
                    size=float(pos_amount),
                    avg_price=float(entry_price),
                )
                await ems_stream.send(ppmsg)
            case _:
                log.warning(
                    'Unhandled event:\n'
                    f'{pformat(msg)}'
                )
--- a/piker/brokers/binance/feed.py
+++ b/piker/brokers/binance/feed.py
@ -1,566 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Real-time and historical data feed endpoints.
 '''
 from __future__ import annotations
 from contextlib import (
    asynccontextmanager as acm,
    aclosing,
 )
 from datetime import datetime
 from functools import (
    partial,
 )
 import itertools
 from pprint import pformat
 from typing import (
    Any,
    AsyncGenerator,
    Callable,
    Generator,
 )
 import time
 import trio
 from trio_typing import TaskStatus
 from pendulum import (
    from_timestamp,
 )
 import numpy as np
 import tractor
 from piker.brokers import (
    open_cached_client,
    NoData,
 )
 from piker._cacheables import (
    async_lifo_cache,
 )
 from piker.accounting import (
    Asset,
    DerivTypes,
    MktPair,
    unpack_fqme,
 )
 from piker.types import Struct
 from piker.data.validate import FeedInit
 from piker.data._web_bs import (
    open_autorecon_ws,
    NoBsWs,
 )
 from piker.log import get_logger
 from piker.brokers._util import (
    DataUnavailable,
 )
 from .api import (
    Client,
 )
 from .venues import (
    Pair,
    FutesPair,
    get_api_eps,
 )
 log = get_logger(name=__name__)
 class L1(Struct):
    # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
    update_id: int
    sym: str
    bid: float
    bsize: float
    ask: float
    asize: float
 # validation type
 # https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Aggregate-Trade-Streams#response-example
 class AggTrade(Struct, frozen=True):
    e: str  # Event type
    E: int  # Event time
    s: str  # Symbol
    a: int  # Aggregate trade ID
    p: float  # Price
    q: float  # Quantity with all the market trades
    f: int  # First trade ID
    l: int  # noqa Last trade ID
    T: int  # Trade time
    m: bool  # Is the buyer the market maker?
    M: bool|None = None  # Ignore
    nq: float|None = None # Normal quantity without the trades involving RPI orders
    # ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
 async def stream_messages(
    ws: NoBsWs,
 ) -> AsyncGenerator[NoBsWs, dict]:
    # TODO: match syntax here!
    msg: dict[str, Any]
    async for msg in ws:
        match msg:
            # for l1 streams binance doesn't add an event type field so
            # identify those messages by matching keys
            # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
            case {
                # NOTE: this is never an old value it seems, so
                # they are always sending real L1 spread updates.
                'u': upid,  # update id
                's': sym,
                'b': bid,
                'B': bsize,
                'a': ask,
                'A': asize,
            }:
                # TODO: it would be super nice to have a `L1` piker type
                # which "renders" incremental tick updates from a packed
                # msg-struct:
                # - backend msgs after packed into the type such that we
                #   can reduce IPC usage but without each backend having
                #   to do that incremental update logic manually B)
                # - would it maybe be more efficient to use this instead?
                #   https://binance-docs.github.io/apidocs/spot/en/#diff-depth-stream
                l1 = L1(
                    update_id=upid,
                    sym=sym,
                    bid=bid,
                    bsize=bsize,
                    ask=ask,
                    asize=asize,
                )
                # for speed probably better to only specifically
                # cast fields we need in numerical form?
                # l1.typecast()
                # repack into piker's tick-quote format
                yield 'l1', {
                    'symbol': l1.sym,
                    'ticks': [
                        {
                            'type': 'bid',
                            'price': float(l1.bid),
                            'size': float(l1.bsize),
                        },
                        {
                            'type': 'bsize',
                            'price': float(l1.bid),
                            'size': float(l1.bsize),
                        },
                        {
                            'type': 'ask',
                            'price': float(l1.ask),
                            'size': float(l1.asize),
                        },
                        {
                            'type': 'asize',
                            'price': float(l1.ask),
                            'size': float(l1.asize),
                        }
                    ]
                }
            # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
            case {
                'e': 'aggTrade',
            }:
                # NOTE: this is purely for a definition,
                # ``msgspec.Struct`` does not runtime-validate until you
                # decode/encode, see:
                # https://jcristharif.com/msgspec/structs.html#type-validation
                msg = AggTrade(**msg)  # TODO: should we .copy() ?
                piker_quote: dict = {
                    'symbol': msg.s,
                    'last': float(msg.p),
                    'brokerd_ts': time.time(),
                    'ticks': [{
                        'type': 'trade',
                        'price': float(msg.p),
                        'size': float(msg.q),
                        'broker_ts': msg.T,
                    }],
                }
                yield 'trade', piker_quote
 def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
    '''
    Create a request subscription packet dict.
    - spot:
      https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
    - futes:
      https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
    '''
    return {
        'method': 'SUBSCRIBE',
        'params': [
            f'{pair.lower()}@{sub_name}'
            for pair in pairs
        ],
        'id': uid
    }
 # TODO, why aren't frame resp `log.info()`s showing in upstream
 # code?!
@acm
 async def open_history_client(
    mkt: MktPair,
 ) -> tuple[Callable, int]:
    # TODO implement history getter for the new storage layer.
    async with open_cached_client('binance') as client:
        async def get_ohlc(
            timeframe: float,
            end_dt: datetime|None = None,
            start_dt: datetime|None = None,
        ) -> tuple[
            np.ndarray,
            datetime,  # start
            datetime,  # end
        ]:
            if timeframe != 60:
                raise DataUnavailable('Only 1m bars are supported')
            # TODO: better wrapping for venue / mode?
            # - eventually logic for usd vs. coin settled futes
            #   based on `MktPair.src` type/value?
            # - maybe something like `async with
            # Client.use_venue('usdtm_futes')`
            if mkt.type_key in DerivTypes:
                client.mkt_mode = 'usdtm_futes'
            else:
                client.mkt_mode = 'spot'
            array: np.ndarray = await client.bars(
                mkt=mkt,
                start_dt=start_dt,
                end_dt=end_dt,
            )
            if array.size == 0:
                raise NoData(
                    f'No frame for {start_dt} -> {end_dt}\n'
                )
            times = array['time']
            if not times.any():
                raise ValueError(
                    'Bad frame with null-times?\n\n'
                    f'{times}'
                )
            # XXX, debug any case where the latest 1m bar we get is
            # already another "sample's-step-old"..
            if end_dt is None:
                inow: int = round(time.time())
                if (
                    _time_step := (inow - times[-1])
                    >
                    timeframe * 2
                ):
                    await tractor.pause()
            start_dt = from_timestamp(times[0])
            end_dt = from_timestamp(times[-1])
            return array, start_dt, end_dt
        yield get_ohlc, {'erlangs': 3, 'rate': 3}
@async_lifo_cache()
 async def get_mkt_info(
    fqme: str,
 ) -> tuple[MktPair, Pair]|None:
    # uppercase since kraken bs_mktid is always upper
    if 'binance' not in fqme.lower():
        fqme += '.binance'
    mkt_mode: str = ''
    broker, mkt_ep, venue, expiry = unpack_fqme(fqme)
    # NOTE: we always upper case all tokens to be consistent with
    # binance's symbology style for pairs, like `BTCUSDT`, but in
    # theory we could also just keep things lower case; as long as
    # we're consistent and the symcache matches whatever this func
    # returns, always!
    expiry: str = expiry.upper()
    venue: str = venue.upper()
    venue_lower: str = venue.lower()
    # XXX TODO: we should change the usdtm_futes name to just
    # usdm_futes (dropping the tether part) since it turns out that
    # there are indeed USD-tokens OTHER THEN tether being used as
    # the margin assets.. it's going to require a wholesale
    # (variable/key) rename as well as file name adjustments to any
    # existing tsdb set..
    if 'usd' in venue_lower:
        mkt_mode: str = 'usdtm_futes'
    # NO IDEA what these contracts (some kinda DEX-ish futes?) are
    # but we're masking them for now..
    elif (
        'defi' in venue_lower
        # TODO: handle coinm futes which have a margin asset that
        # is some crypto token!
        # https://binance-docs.github.io/apidocs/delivery/en/#exchange-information
        or 'btc' in venue_lower
    ):
        return None
    else:
        # NOTE: see the `FutesPair.bs_fqme: str` implementation
        # to understand the reverse market info lookup below.
        mkt_mode = venue_lower or 'spot'
    if (
        venue
        and 'spot' not in venue_lower
        # XXX: catch all in case user doesn't know which
        # venue they want (usdtm vs. coinm) and we can choose
        # a default (via config?) once we support coin-m APIs.
        or 'perp' in venue_lower
    ):
        if not mkt_mode:
            mkt_mode: str = f'{venue_lower}_futes'
    async with open_cached_client(
        'binance',
    ) as client:
        assets: dict[str, Asset] = await client.get_assets()
        pair_str: str = mkt_ep.upper()
        # switch venue-mode depending on input pattern parsing
        # since we want to use a particular endpoint (set) for
        # pair info lookup!
        client.mkt_mode = mkt_mode
        pair: Pair = await client.exch_info(
            pair_str,
            venue=mkt_mode,  # explicit
            expiry=expiry,
        )
        if 'futes' in mkt_mode:
            assert isinstance(pair, FutesPair)
        dst: Asset|None = assets.get(pair.bs_dst_asset)
        if (
            not dst
            # TODO: a known asset DNE list?
            # and pair.baseAsset == 'DEFI'
        ):
            log.warning(
                f'UNKNOWN {venue} asset {pair.baseAsset} from,\n'
                f'{pformat(pair.to_dict())}'
            )
            # XXX UNKNOWN missing "asset", though no idea why?
            # maybe it's only avail in the margin venue(s): /dapi/ ?
            return None
        mkt = MktPair(
            dst=dst,
            src=assets[pair.bs_src_asset],
            price_tick=pair.price_tick,
            size_tick=pair.size_tick,
            bs_mktid=pair.symbol,
            expiry=expiry,
            venue=venue,
            broker='binance',
            # NOTE: sectype is always taken from dst, see
            # `MktPair.type_key` and `Client._cache_pairs()`
            # _atype=sectype,
        )
        return mkt, pair
@acm
 async def subscribe(
    ws: NoBsWs,
    symbols: list[str],
    # defined once at import time to keep a global state B)
    iter_subids: Generator[int, None, None] = itertools.count(),
 ):
    # setup subs
    subid: int = next(iter_subids)
    # trade data (aka L1)
    # https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
    l1_sub = make_sub(symbols, 'bookTicker', subid)
    await ws.send_msg(l1_sub)
    # aggregate (each order clear by taker **not** by maker)
    # trades data:
    # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
    agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
    await ws.send_msg(agg_trades_sub)
    # might get ack from ws server, or maybe some
    # other msg still in transit..
    res = await ws.recv_msg()
    subid: str|None = res.get('id')
    if subid:
        assert res['id'] == subid
    yield
    subs = []
    for sym in symbols:
        subs.append("{sym}@aggTrade")
        subs.append("{sym}@bookTicker")
    # unsub from all pairs on teardown
    if ws.connected():
        await ws.send_msg({
            "method": "UNSUBSCRIBE",
            "params": subs,
            "id": subid,
        })
        # XXX: do we need to ack the unsub?
        # await ws.recv_msg()
 async def stream_quotes(
    send_chan: trio.abc.SendChannel,
    symbols: list[str],
    feed_is_live: trio.Event,
    loglevel: str = None,
    # startup sync
    task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
 ) -> None:
    async with (
        tractor.trionics.maybe_raise_from_masking_exc(),
        send_chan as send_chan,
        open_cached_client('binance') as client,
    ):
        init_msgs: list[FeedInit] = []
        for sym in symbols:
            mkt: MktPair
            pair: Pair
            mkt, pair = await get_mkt_info(sym)
            # build out init msgs according to latest spec
            init_msgs.append(
                FeedInit(mkt_info=mkt)
            )
        wss_url: str = get_api_eps(client.mkt_mode)[1]  # 2nd elem is wss url
        # TODO: for sanity, but remove eventually Xp
        if 'future' in mkt.type_key:
            assert 'fstream' in wss_url
        async with (
            open_autorecon_ws(
                url=wss_url,
                fixture=partial(
                    subscribe,
                    symbols=[mkt.bs_mktid],
                ),
            ) as ws,
            # avoid stream-gen closure from breaking trio..
            aclosing(stream_messages(ws)) as msg_gen,
        ):
            # log.info('WAITING ON FIRST LIVE QUOTE..')
            typ, quote = await anext(msg_gen)
            # pull a first quote and deliver
            while typ != 'trade':
                typ, quote = await anext(msg_gen)
            task_status.started((init_msgs, quote))
            # signal to caller feed is ready for consumption
            feed_is_live.set()
            # import time
            # last = time.time()
            # XXX NOTE: can't include the `.binance` suffix
            # or the sampling loop will not broadcast correctly
            # since `bus._subscribers.setdefault(bs_fqme, set())`
            # is used inside `.data.open_feed_bus()` !!!
            topic: str = mkt.bs_fqme
            # start streaming
            async for typ, quote in msg_gen:
                # period = time.time() - last
                # hz = 1/period if period else float('inf')
                # if hz > 60:
                #     log.info(f'Binance quotez : {hz}')
                await send_chan.send({topic: quote})
                # last = time.time()
@tractor.context
 async def open_symbol_search(
    ctx: tractor.Context,
 ) -> Client:
    # NOTE: symbology tables are loaded as part of client
    # startup in ``.api.get_client()`` and in this case
    # are stored as `Client._pairs`.
    async with open_cached_client('binance') as client:
        # TODO: maybe we should deliver the cache
        # so that client's can always do a local-lookup-first
        # style try and then update async as (new) match results
        # are delivered from here?
        await ctx.started()
        async with ctx.open_stream() as stream:
            pattern: str
            async for pattern in stream:
                # NOTE: pattern fuzzy-matching is done within
                # the methd impl.
                pairs: dict[str, Pair] = await client.search_symbols(
                    pattern,
                )
                # repack in fqme-keyed table
                byfqme: dict[str, Pair] = {}
                for pair in pairs.values():
                    byfqme[pair.bs_fqme] = pair
                await stream.send(byfqme)
--- a/piker/brokers/binance/venues.py
+++ b/piker/brokers/binance/venues.py
@ -1,323 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 Per market data-type definitions and schemas types.
 """
 from __future__ import annotations
 from typing import (
    Literal,
 )
 from decimal import Decimal
 from msgspec import field
 from piker.types import Struct
 # API endpoint paths by venue / sub-API
 _domain: str = 'binance.com'
 _spot_url = f'https://api.{_domain}'
 _futes_url = f'https://fapi.{_domain}'
 # WEBsocketz
 # NOTE XXX: see api docs which show diff addr?
 # https://developers.binance.com/docs/binance-trading-api/websocket_api#general-api-information
 _spot_ws: str = 'wss://stream.binance.com/ws'
 # or this one? ..
 # 'wss://ws-api.binance.com:443/ws-api/v3',
 # https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
 _futes_ws: str = f'wss://fstream.{_domain}/ws'
 _auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
 # test nets
 # NOTE: spot test network only allows certain ep sets:
 # https://testnet.binance.vision/
 # https://www.binance.com/en/support/faq/how-to-test-my-functions-on-binance-testnet-ab78f9a1b8824cf0a106b4229c76496d
 _testnet_spot_url: str = 'https://testnet.binance.vision/api'
 _testnet_spot_ws: str = 'wss://testnet.binance.vision/ws'
 # or this one? ..
 # 'wss://testnet.binance.vision/ws-api/v3'
 _testnet_futes_url: str = 'https://testnet.binancefuture.com'
 _testnet_futes_ws: str = 'wss://stream.binancefuture.com/ws'
 MarketType = Literal[
    'spot',
    # 'margin',
    'usdtm_futes',
    # 'coinm_futes',
 ]
 def get_api_eps(venue: MarketType) -> tuple[str, str]:
    '''
    Return API ep root paths per venue.
    '''
    return {
        'spot': (
            _spot_url,
            _spot_ws,
        ),
        'usdtm_futes': (
            _futes_url,
            _futes_ws,
        ),
    }[venue]
 class Pair(Struct, frozen=True, kw_only=True):
    symbol: str
    status: str
    orderTypes: list[str]
    # src
    quoteAsset: str
    quotePrecision: int
    # dst
    baseAsset: str
    baseAssetPrecision: int
    permissionSets: list[list[str]]
    # https://developers.binance.com/docs/binance-spot-api-docs#2025-08-26
    # will become non-optional 2025-08-28?
    # https://developers.binance.com/docs/binance-spot-api-docs#future-changes
    pegInstructionsAllowed: bool = False
    # https://developers.binance.com/docs/binance-spot-api-docs#2025-12-02
    opoAllowed: bool = False
    filters: dict[
        str,
        str | int | float,
    ] = field(default_factory=dict)
    @property
    def price_tick(self) -> Decimal:
        # XXX: lul, after manually inspecting the response format we
        # just directly pick out the info we need
        step_size: str = self.filters['PRICE_FILTER']['tickSize'].rstrip('0')
        return Decimal(step_size)
    @property
    def size_tick(self) -> Decimal:
        step_size: str = self.filters['LOT_SIZE']['stepSize'].rstrip('0')
        return Decimal(step_size)
    @property
    def bs_fqme(self) -> str:
        return self.symbol
    @property
    def bs_mktid(self) -> str:
        return f'{self.symbol}.{self.venue}'
 class SpotPair(Pair, frozen=True):
    cancelReplaceAllowed: bool
    allowTrailingStop: bool
    quoteAssetPrecision: int
    baseCommissionPrecision: int
    quoteCommissionPrecision: int
    icebergAllowed: bool
    ocoAllowed: bool
    quoteOrderQtyMarketAllowed: bool
    isSpotTradingAllowed: bool
    isMarginTradingAllowed: bool
    otoAllowed: bool
    defaultSelfTradePreventionMode: str
    allowedSelfTradePreventionModes: list[str]
    permissions: list[str]
    # can the paint botz creat liq gaps even easier on this asset?
    # Bp
    # https://developers.binance.com/docs/binance-spot-api-docs/faqs/order_amend_keep_priority
    amendAllowed: bool
    # NOTE: see `.data._symcache.SymbologyCache.load()` for why
    ns_path: str = 'piker.brokers.binance:SpotPair'
    @property
    def venue(self) -> str:
        return 'SPOT'
    @property
    def bs_fqme(self) -> str:
        return f'{self.symbol}.SPOT'
    @property
    def bs_src_asset(self) -> str:
        return f'{self.quoteAsset}'
    @property
    def bs_dst_asset(self) -> str:
        return f'{self.baseAsset}'
 class FutesPair(Pair):
    symbol: str  # 'BTCUSDT',
    pair: str  # 'BTCUSDT',
    baseAssetPrecision: int # 8,
    contractType: str  # 'PERPETUAL',
    deliveryDate: int   # 4133404800000,
    liquidationFee: float  # '0.012500',
    maintMarginPercent: float  # '2.5000',
    marginAsset: str  # 'USDT',
    marketTakeBound: float  # '0.05',
    maxMoveOrderLimit: int  # 10000,
    onboardDate: int  # 1569398400000,
    pricePrecision: int  # 2,
    quantityPrecision: int  # 3,
    quoteAsset: str  # 'USDT',
    quotePrecision: int  # 8,
    requiredMarginPercent: float  # '5.0000',
    timeInForce: list[str]  # ['GTC', 'IOC', 'FOK', 'GTX'],
    triggerProtect: float  # '0.0500',
    underlyingSubType: list[str]  # ['PoW'],
    underlyingType: str  # 'COIN'
    # NOTE: see `.data._symcache.SymbologyCache.load()` for why
    ns_path: str = 'piker.brokers.binance:FutesPair'
    # NOTE: for compat with spot pairs and `MktPair.src: Asset`
    # processing..
    @property
    def quoteAssetPrecision(self) -> int:
        return self.quotePrecision
    @property
    def expiry(self) -> str:
        symbol: str = self.symbol
        contype: str = self.contractType
        match contype:
            case (
                'CURRENT_QUARTER'
                | 'CURRENT_QUARTER DELIVERING'
                | 'NEXT_QUARTER'  # su madre binance..
            ):
                pair, _, expiry = symbol.partition('_')
                assert pair == self.pair  # sanity
                return f'{expiry}'
            case (
                'PERPETUAL'
                | 'TRADIFI_PERPETUAL'
            ):
                return 'PERP'
            case '':
                subtype: list[str] = self.underlyingSubType
                if not subtype:
                    if self.status == 'PENDING_TRADING':
                        return 'PENDING'
                match subtype:
                    case ['DEFI']:
                        return 'PERP'
        # wow, just wow you binance guys suck..
        if self.status == 'PENDING_TRADING':
            return 'PENDING'
        # XXX: yeah no clue then..
        raise ValueError(
            f'Bad .expiry token match: {contype} for {symbol}'
        )
    @property
    def venue(self) -> str:
        symbol: str = self.symbol
        ctype: str = self.contractType
        margin: str = self.marginAsset
        match ctype:
            case (
                'PERPETUAL'
                | 'TRADIFI_PERPETUAL'
            ):
                return f'{margin}M'
            case (
                'CURRENT_QUARTER'
                | 'CURRENT_QUARTER DELIVERING'
                | 'NEXT_QUARTER'  # su madre binance..
            ):
                _, _, expiry = symbol.partition('_')
                return f'{margin}M'
            case '':
                subtype: list[str] = self.underlyingSubType
                if not subtype:
                    if self.status == 'PENDING_TRADING':
                        return f'{margin}M'
                match subtype:
                    case (
                        ['DEFI']
                        | ['USDC']
                    ):
                        return f'{subtype[0]}'
        # XXX: yeah no clue then..
        raise ValueError(
            f'Bad .venue token match: {ctype}'
        )
    @property
    def bs_fqme(self) -> str:
        symbol: str = self.symbol
        ctype: str = self.contractType
        venue: str = self.venue
        pair: str = self.pair
        match ctype:
            case (
                'CURRENT_QUARTER'
                | 'NEXT_QUARTER'  # su madre binance..
            ):
                pair, _, expiry = symbol.partition('_')
                assert pair == self.pair
        return f'{pair}.{venue}.{self.expiry}'
    @property
    def bs_src_asset(self) -> str:
        return f'{self.quoteAsset}'
    @property
    def bs_dst_asset(self) -> str:
        return f'{self.baseAsset}.{self.venue}'
 PAIRTYPES: dict[MarketType, Pair] = {
    'spot': SpotPair,
    'usdtm_futes': FutesPair,
    # TODO: support coin-margined venue:
    # https://binance-docs.github.io/apidocs/delivery/en/#change-log
    # 'coinm_futes': CoinFutesPair,
 }
--- a/piker/brokers/cli.py
+++ b/piker/brokers/cli.py
@ -21,37 +21,24 @@ import os
 from functools import partial
 from operator import attrgetter
 from operator import itemgetter
 from types import ModuleType
 import click
 import trio
 import tractor
-from piker.cli import cli
+from ..cli import cli
-from piker import watchlists as wl
+from .. import watchlists as wl
-from piker.log import (
+from ..log import get_console_log, colorize_json, get_logger
-    colorize_json,
+from .._daemon import maybe_spawn_brokerd, maybe_open_pikerd
-    get_console_log,
+from ..brokers import core, get_brokermod, data
    get_logger,
 )
 from ..service import (
    maybe_spawn_brokerd,
    maybe_open_pikerd,
 )
 from ..brokers import (
    core,
    get_brokermod,
    data,
 )
-log = get_logger(
+log = get_logger('cli')
-    name=__name__,
+DEFAULT_BROKER = 'questrade'
 )
 DEFAULT_BROKER = 'binance'
 _config_dir = click.get_app_dir('piker')
 _watchlists_data_path = os.path.join(_config_dir, 'watchlists.json')
 OK = '\033[92m'
 WARNING = '\033[93m'
 FAIL = '\033[91m'
@ -73,7 +60,6 @@ def get_method(client, meth_name: str):
    print_ok('found!.')
    return method
 async def run_method(client, meth_name: str, **kwargs):
    method = get_method(client, meth_name)
    print('running...', end='', flush=True)
@ -81,20 +67,19 @@ async def run_method(client, meth_name: str, **kwargs):
    print_ok(f'done! result: {type(result)}')
    return result
 async def run_test(broker_name: str):
    brokermod = get_brokermod(broker_name)
    total = 0
    passed = 0
    failed = 0
-    print('getting client...', end='', flush=True)
+    print(f'getting client...', end='', flush=True)
    if not hasattr(brokermod, 'get_client'):
        print_error('fail! no \'get_client\' context manager found.')
        return
    async with brokermod.get_client(is_brokercheck=True) as client:
-        print_ok('done! inside client context.')
+        print_ok(f'done! inside client context.')
        # check for methods present on brokermod
        method_list = [
@ -145,6 +130,7 @@ async def run_test(broker_name: str):
            total += 1
        # check for methods present con brokermod.Client and their
        # results
@ -194,9 +180,10 @@ def brokercheck(config, broker):
    trio.run(run_test, broker)
@cli.command()
@click.option('--keys', '-k', multiple=True,
-            help='Return results only for these keys')
+              help='Return results only for these keys')
@click.argument('meth', nargs=1)
@click.argument('kwargs', nargs=-1)
@click.pass_obj
@ -243,7 +230,7 @@ def quote(config, tickers):
    '''
    # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
    quotes = trio.run(partial(core.stocks_quote, brokermod, tickers))
    if not quotes:
@ -270,7 +257,7 @@ def bars(config, symbol, count):
    '''
    # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
    # broker backend should return at the least a
    # list of candle dictionaries
@ -305,7 +292,7 @@ def record(config, rate, name, dhost, filename):
    '''
    # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
    loglevel = config['loglevel']
    log = config['log']
@ -346,10 +333,9 @@ def contracts(ctx, loglevel, broker, symbol, ids):
    '''
    brokermod = get_brokermod(broker)
-    get_console_log(
+    get_console_log(loglevel)
-        level=loglevel,
+
-        name=__name__,
+
    )
    contracts = trio.run(partial(core.contracts, brokermod, symbol))
    if not ids:
@ -373,7 +359,7 @@ def optsquote(config, symbol, date):
    '''
    # global opts
-    brokermod = list(config['brokermods'].values())[0]
+    brokermod = config['brokermods'][0]
    quotes = trio.run(
        partial(
@ -390,157 +376,58 @@ def optsquote(config, symbol, date):
@cli.command()
@click.argument('tickers', nargs=-1, required=True)
@click.pass_obj
-def mkt_info(
+def symbol_info(config, tickers):
    config: dict,
    tickers: list[str],
 ):
    '''
    Print symbol quotes to the console
    '''
    from msgspec.json import encode, decode
    from ..accounting import MktPair
    from ..service import (
        open_piker_runtime,
    )
    # global opts
-    brokermods: dict[str, ModuleType] = config['brokermods']
+    brokermod = config['brokermods'][0]
-    mkts: list[MktPair] = []
+    quotes = trio.run(partial(core.symbol_info, brokermod, tickers))
-    async def main():
+    if not quotes:
-
+        log.error(f"No quotes could be found for {tickers}?")
        async with open_piker_runtime(
            name='mkt_info_query',
            # loglevel=loglevel,
            debug_mode=True,
        ) as (_, _):
            for fqme in tickers:
                bs_fqme, _, broker = fqme.rpartition('.')
                brokermod: ModuleType = brokermods[broker]
                mkt, bs_pair = await core.mkt_info(
                    brokermod,
                    bs_fqme,
                )
                mkts.append((mkt, bs_pair))
    trio.run(main)
    if not mkts:
        log.error(
            f'No market info could be found for {tickers}'
        )
        return
-    if len(mkts) < len(tickers):
+    if len(quotes) < len(tickers):
-        syms = tuple(map(itemgetter('fqme'), mkts))
+        syms = tuple(map(itemgetter('symbol'), quotes))
        for ticker in tickers:
            if ticker not in syms:
-                log.warn(f"Could not find symbol {ticker}?")
+                brokermod.log.warn(f"Could not find symbol {ticker}?")
-
+    click.echo(colorize_json(quotes))
    # TODO: use ``rich.Table`` intead here!
    for mkt, bs_pair in mkts:
        click.echo(
            '\n'
            '----------------------------------------------------\n'
            f'{type(bs_pair)}\n'
            '----------------------------------------------------\n'
            f'{colorize_json(bs_pair.to_dict())}\n'
            '----------------------------------------------------\n'
            f'as piker `MktPair` with fqme: {mkt.fqme}\n'
            '----------------------------------------------------\n'
            # NOTE: roundtrip to json codec for console print
            f'{colorize_json(decode(encode(mkt)))}'
        )
@cli.command()
@click.argument('pattern', required=True)
 # TODO: move this to top level click/typer context for all subs
@click.option(
    '--pdb',
    is_flag=True,
    help='Enable tractor debug mode',
 )
@click.pass_obj
-def search(
+def search(config, pattern):
    config: dict,
    pattern: str,
    pdb: bool,
 ):
    '''
    Search for symbols from broker backend(s).
    '''
    # global opts
-    brokermods: list[ModuleType] = list(config['brokermods'].values())
+    brokermods = config['brokermods']
    # TODO: this is coming from the `search --pdb` NOT from
    # the `piker --pdb` XD ..
    # -[ ] pull from the parent click ctx's values..dumdum
    # assert pdb
    loglevel: str = config['loglevel']
    # define tractor entrypoint
    async def main(func):
        async with maybe_open_pikerd(
-            loglevel=loglevel,
+            loglevel=config['loglevel'],
            debug_mode=pdb,
        ):
            return await func()
-    from piker.toolz import open_crash_handler
+    quotes = trio.run(
-    with open_crash_handler():
+        main,
-        quotes = trio.run(
+        partial(
-            main,
+            core.symbol_search,
-            partial(
+            brokermods,
-                core.symbol_search,
+            pattern,
-                brokermods,
+        ),
-                pattern,
+    )
                loglevel=loglevel,
            ),
        )
-        if not quotes:
+    if not quotes:
-            log.error(f"No matches could be found for {pattern}?")
+        log.error(f"No matches could be found for {pattern}?")
-            return
+        return
-        click.echo(colorize_json(quotes))
+    click.echo(colorize_json(quotes))
@cli.command()
@click.argument('section', required=False)
@click.argument('value', required=False)
@click.option('--delete', '-d', flag_value=True, help='Delete section')
@click.pass_obj
 def brokercfg(config, section, value, delete):
    '''
    If invoked with no arguments, open an editor to edit broker
    configs file or get / update an individual section.
    '''
    from .. import config
    if section:
        conf, path = config.load()
        if not delete:
            if value:
                config.set_value(conf, section, value)
            click.echo(
                colorize_json(
                    config.get_value(conf, section))
            )
        else:
            config.del_value(conf, section)
        config.write(config=conf)
    else:
        conf, path = config.load(raw=True)
        config.write(
            raw=click.edit(text=conf)
        )
--- a/piker/brokers/core.py
+++ b/piker/brokers/core.py
@ -22,26 +22,22 @@ routines should be primitive data types where possible.
 """
 import inspect
 from types import ModuleType
-from typing import (
+from typing import List, Dict, Any, Optional
    Any,
 )
 import trio
-from piker.log import get_logger
+from ..log import get_logger
 from . import get_brokermod
-from ..service import maybe_spawn_brokerd
+from .._daemon import maybe_spawn_brokerd
-from . import open_cached_client
+from .._cacheables import open_cached_client
 from ..accounting import MktPair
-log = get_logger(name=__name__)
+
 log = get_logger(__name__)
 async def api(brokername: str, methname: str, **kwargs) -> dict:
-    '''
+    """Make (proxy through) a broker API call by name and return its result.
-    Make (proxy through) a broker API call by name and return its result.
+    """
    '''
    brokermod = get_brokermod(brokername)
    async with brokermod.get_client() as client:
        meth = getattr(client, methname, None)
@ -68,14 +64,10 @@ async def api(brokername: str, methname: str, **kwargs) -> dict:
 async def stocks_quote(
    brokermod: ModuleType,
-    tickers: list[str]
+    tickers: List[str]
-
+) -> Dict[str, Dict[str, Any]]:
-) -> dict[str, dict[str, Any]]:
+    """Return quotes dict for ``tickers``.
-    '''
+    """
    Return a `dict` of snapshot quotes for the provided input
    `tickers`: a `list` of fqmes.
    '''
    async with brokermod.get_client() as client:
        return await client.quote(tickers)
@ -84,15 +76,13 @@ async def stocks_quote(
 async def option_chain(
    brokermod: ModuleType,
    symbol: str,
-    date: str|None = None,
+    date: Optional[str] = None,
-) -> dict[str, dict[str, dict[str, Any]]]:
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
-    '''
+    """Return option chain for ``symbol`` for ``date``.
    Return option chain for ``symbol`` for ``date``.
    By default all expiries are returned. If ``date`` is provided
    then contract quotes for that single expiry are returned.
-
+    """
    '''
    async with brokermod.get_client() as client:
        if date:
            id = int((await client.tickers2ids([symbol]))[symbol])
@ -107,39 +97,41 @@ async def option_chain(
            return await client.option_chains(contracts)
-# async def contracts(
+async def contracts(
-#     brokermod: ModuleType,
+    brokermod: ModuleType,
-#     symbol: str,
+    symbol: str,
-# ) -> dict[str, dict[str, dict[str, Any]]]:
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
-#     """Return option contracts (all expiries) for ``symbol``.
+    """Return option contracts (all expiries) for ``symbol``.
-#     """
+    """
-#     async with brokermod.get_client() as client:
+    async with brokermod.get_client() as client:
-#         # return await client.get_all_contracts([symbol])
+        # return await client.get_all_contracts([symbol])
-#         return await client.get_all_contracts([symbol])
+        return await client.get_all_contracts([symbol])
 async def bars(
    brokermod: ModuleType,
    symbol: str,
    **kwargs,
-) -> dict[str, dict[str, dict[str, Any]]]:
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
-    '''
+    """Return option contracts (all expiries) for ``symbol``.
-    Return option contracts (all expiries) for ``symbol``.
+    """
    '''
    async with brokermod.get_client() as client:
        return await client.bars(symbol, **kwargs)
-async def search_w_brokerd(
+async def symbol_info(
-    name: str,
+    brokermod: ModuleType,
-    pattern: str,
+    symbol: str,
-) -> dict:
+    **kwargs,
 ) -> Dict[str, Dict[str, Dict[str, Any]]]:
    """Return symbol info from broker.
    """
    async with brokermod.get_client() as client:
        return await client.symbol_info(symbol, **kwargs)
 async def search_w_brokerd(name: str, pattern: str) -> dict:
    # TODO: WHY NOT WORK!?!
    # when we `step` through the next block?
    # import tractor
    # await tractor.pause()
    async with open_cached_client(name) as client:
        # TODO: support multiple asset type concurrent searches.
@ -149,15 +141,14 @@ async def search_w_brokerd(
 async def symbol_search(
    brokermods: list[ModuleType],
    pattern: str,
    loglevel: str = 'warning',
    **kwargs,
-) -> dict[str, dict[str, dict[str, Any]]]:
+) -> Dict[str, Dict[str, Dict[str, Any]]]:
    '''
    Return symbol info from broker.
    '''
-    results: list[str] = []
+    results = []
    async def search_backend(
        brokermod: ModuleType
@ -165,21 +156,9 @@ async def symbol_search(
        brokername: str = mod.name
        # TODO: figure this the FUCK OUT
        # -> ok so obvi in the root actor any async task that's
        # spawned outside the main tractor-root-actor task needs to
        # call this..
        # await tractor.devx._debug.maybe_init_greenback()
        # tractor.pause_from_sync()
        async with maybe_spawn_brokerd(
            mod.name,
-            infect_asyncio=getattr(
+            infect_asyncio=getattr(mod, '_infect_asyncio', False),
                mod,
                '_infect_asyncio',
                False,
            ),
            loglevel=loglevel
        ) as portal:
            results.append((
@ -192,26 +171,8 @@ async def symbol_search(
            ))
    async with trio.open_nursery() as n:
        for mod in brokermods:
            n.start_soon(search_backend, mod.name)
    return results
 async def mkt_info(
    brokermod: ModuleType,
    fqme: str,
    **kwargs,
 ) -> MktPair:
    '''
    Return the `piker.accounting.MktPair` info struct from a given
    backend broker tradable src/dst asset pair.
    '''
    async with open_cached_client(brokermod.name) as client:
        assert client
        return await brokermod.get_mkt_info(
            fqme.replace(brokermod.name, '')
        )
--- a/piker/brokers/data.py
+++ b/piker/brokers/data.py
@ -41,15 +41,12 @@ import tractor
 from tractor.experimental import msgpub
 from async_generator import asynccontextmanager
-from piker.log import(
+from ..log import get_logger, get_console_log
    get_logger,
    get_console_log,
 )
 from . import get_brokermod
-log = get_logger(
+
-    name='piker.brokers.binance',
+log = get_logger(__name__)
-)
+
 async def wait_for_network(
    net_func: Callable,
@ -230,31 +227,26 @@ async def get_cached_feed(
@tractor.stream
 async def start_quote_stream(
-    stream: tractor.Context,  # marks this as a streaming func
+    ctx: tractor.Context,  # marks this as a streaming func
    broker: str,
    symbols: List[Any],
    feed_type: str = 'stock',
    rate: int = 3,
 ) -> None:
-    '''
+    """Handle per-broker quote stream subscriptions using a "lazy" pub-sub
    Handle per-broker quote stream subscriptions using a "lazy" pub-sub
    pattern.
    Spawns new quoter tasks for each broker backend on-demand.
    Since most brokers seems to support batch quote requests we
    limit to one task per process (for now).
-
+    """
    '''
    # XXX: why do we need this again?
-    get_console_log(
+    get_console_log(tractor.current_actor().loglevel)
        level=tractor.current_actor().loglevel,
        name=__name__,
    )
    # pull global vars from local actor
    symbols = list(symbols)
    log.info(
-        f"{stream.chan.uid} subscribed to {broker} for symbols {symbols}")
+        f"{ctx.chan.uid} subscribed to {broker} for symbols {symbols}")
    # another actor task may have already created it
    async with get_cached_feed(broker) as feed:
@ -298,13 +290,13 @@ async def start_quote_stream(
            assert fquote['displayable']
            payload[sym] = fquote
-        await stream.send_yield(payload)
+        await ctx.send_yield(payload)
        await stream_poll_requests(
            # ``trionics.msgpub`` required kwargs
            task_name=feed_type,
-            ctx=stream,
+            ctx=ctx,
            topics=symbols,
            packetizer=feed.mod.packetizer,
@ -327,11 +319,9 @@ async def call_client(
 class DataFeed:
-    '''
+    """Data feed client for streaming symbol data from and making API client calls
-    Data feed client for streaming symbol data from and making API
+    to a (remote) ``brokerd`` daemon.
-    client calls to a (remote) ``brokerd`` daemon.
+    """
    '''
    _allowed = ('stock', 'option')
    def __init__(self, portal, brokermod):
--- a/piker/brokers/deribit/init.py
+++ b/piker/brokers/deribit/init.py
@ -21,6 +21,8 @@ Deribit backend.
 from piker.log import get_logger
 log = get_logger(__name__)
 from .api import (
    get_client,
 )
@ -28,15 +30,13 @@ from .feed import (
    open_history_client,
    open_symbol_search,
    stream_quotes,
-    # backfill_bars,
+    backfill_bars
 )
 # from .broker import (
-    # open_trade_dialog,
+    # trades_dialogue,
    # norm_trade_records,
 # )
 log = get_logger(__name__)
 __all__ = [
    'get_client',
 #    'trades_dialogue',
--- a/piker/brokers/deribit/api.py
+++ b/piker/brokers/deribit/api.py
@ -18,34 +18,43 @@
 Deribit backend.
 '''
-import asyncio
+import json
 from contextlib import (
    asynccontextmanager as acm,
 )
 from datetime import datetime
 from functools import partial
 import time
-from typing import (
+import asyncio
-    Any,
+
-    Optional,
+from contextlib import asynccontextmanager as acm, AsyncExitStack
-    Callable,
+from functools import partial
 from datetime import datetime
 from typing import Any, Optional, Iterable, Callable
 import pendulum
 import asks
 import trio
 from trio_typing import Nursery, TaskStatus
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 from piker.data.types import Struct
 from piker.data._web_bs import (
    NoBsWs,
    open_autorecon_ws,
    open_jsonrpc_session
 )
-from pendulum import now
+from .._util import resproc
-import trio
+
-from trio_typing import TaskStatus
+from piker import config
-from rapidfuzz import process as fuzzy
+from piker.log import get_logger
-import numpy as np
+
 from tractor.trionics import (
    broadcast_receiver,
    BroadcastReceiver,
    maybe_open_context
    collapse_eg,
 )
 from tractor import to_asyncio
-# XXX WOOPS XD
+
 # yeah you'll need to install it since it was removed in #489 by
 # accident; well i thought we had removed all usage..
 from cryptofeed import FeedHandler
 from cryptofeed.defines import (
    DERIBIT,
    L1_BOOK, TRADES,
@ -53,20 +62,6 @@ from cryptofeed.defines import (
 )
 from cryptofeed.symbols import Symbol
 from piker.data import (
    def_iohlcv_fields,
    match_from_pairs,
    Struct,
 )
 from piker.data._web_bs import (
    open_jsonrpc_session
 )
 from piker import config
 from piker.log import get_logger
 log = get_logger(__name__)
@ -80,13 +75,26 @@ _ws_url = 'wss://www.deribit.com/ws/api/v2'
 _testnet_ws_url = 'wss://test.deribit.com/ws/api/v2'
 # Broker specific ohlc schema (rest)
 _ohlc_dtype = [
    ('index', int),
    ('time', int),
    ('open', float),
    ('high', float),
    ('low', float),
    ('close', float),
    ('volume', float),
    ('bar_wap', float),  # will be zeroed by sampler if not filled
 ]
 class JSONRPCResult(Struct):
    jsonrpc: str = '2.0'
    id: int
-    result: Optional[list[dict]] = None
+    result: Optional[dict] = None
    error: Optional[dict] = None
-    usIn: int
+    usIn: int 
-    usOut: int
+    usOut: int 
    usDiff: int
    testnet: bool
@ -293,29 +301,24 @@ class Client:
        currency: str = 'btc',  # BTC, ETH, SOL, USDC
        kind: str = 'option',
        expired: bool = False
    ) -> dict[str, Any]:
        """Get symbol info for the exchange.
-    ) -> dict[str, dict]:
+        """
        '''
        Get symbol infos.
        '''
        if self._pairs:
            return self._pairs
        # will retrieve all symbols by default
-        params: dict[str, str] = {
+        params = {
            'currency': currency.upper(),
            'kind': kind,
            'expired': str(expired).lower()
        }
-        resp: JSONRPCResult = await self.json_rpc(
+        resp = await self.json_rpc('public/get_instruments', params)
-            'public/get_instruments',
+        results = resp.result
-            params,
+
-        )
+        instruments = {
        # convert to symbol-keyed table
        results: list[dict] | None = resp.result
        instruments: dict[str, dict] = {
            item['instrument_name'].lower(): item
            for item in results
        }
@ -328,7 +331,6 @@ class Client:
    async def cache_symbols(
        self,
    ) -> dict:
        if not self._pairs:
            self._pairs = await self.symbol_info()
@ -339,23 +341,17 @@ class Client:
        pattern: str,
        limit: int = 30,
    ) -> dict[str, Any]:
-        '''
+        data = await self.symbol_info()
        Fuzzy search symbology set for pairs matching `pattern`.
-        '''
+        matches = fuzzy.extractBests(
-        pairs: dict[str, Any] = await self.symbol_info()
+            pattern,
-        matches: dict[str, Pair] = match_from_pairs(
+            data,
            pairs=pairs,
            query=pattern.upper(),
            score_cutoff=35,
            limit=limit
        )
-
+        # repack in dict form
-       # repack in name-keyed table
+        return {item[0]['instrument_name'].lower(): item[0]
-        return {
+                for item in matches}
            pair['instrument_name'].lower(): pair
            for pair in matches.values()
        }
    async def bars(
        self,
@ -409,7 +405,7 @@ class Client:
            new_bars.append((i,) + tuple(row))
-        array = np.array(new_bars, dtype=def_iohlcv_fields) if as_np else klines
+        array = np.array(new_bars, dtype=_ohlc_dtype) if as_np else klines
        return array
    async def last_trades(
@ -433,7 +429,6 @@ async def get_client(
 ) -> Client:
    async with (
        collapse_eg(),
        trio.open_nursery() as n,
        open_jsonrpc_session(
            _testnet_ws_url, dtype=JSONRPCResult) as json_rpc
--- a/piker/brokers/deribit/feed.py
+++ b/piker/brokers/deribit/feed.py
@ -26,11 +26,11 @@ import time
 import trio
 from trio_typing import TaskStatus
 import pendulum
-from rapidfuzz import process as fuzzy
+from fuzzywuzzy import process as fuzzy
 import numpy as np
 import tractor
-from piker.brokers import open_cached_client
+from piker._cacheables import open_cached_client
 from piker.log import get_logger, get_console_log
 from piker.data import ShmArray
 from piker.brokers._util import (
@ -39,6 +39,7 @@ from piker.brokers._util import (
 )
 from cryptofeed import FeedHandler
 from cryptofeed.defines import (
    DERIBIT, L1_BOOK, TRADES, OPTION, CALL, PUT
 )
@ -61,10 +62,9 @@ log = get_logger(__name__)
@acm
 async def open_history_client(
-    mkt: MktPair,
+    instrument: str,
 ) -> tuple[Callable, int]:
    fnstrument: str = mkt.bs_fqme
    # TODO implement history getter for the new storage layer.
    async with open_cached_client('deribit') as client:
--- a/piker/brokers/ib/README.rst
+++ b/piker/brokers/ib/README.rst
@ -2,7 +2,7 @@
 --------------
 more or less the "everything broker" for traditional and international
 markets. they are the "go to" provider for automatic retail trading
-and we interface to their APIs using the `ib_async` project.
+and we interface to their APIs using the `ib_insync` project.
 status
 ******
@ -127,7 +127,7 @@ your ``pps.toml`` file will have position entries like,
    [ib.algopaper."mnq.globex.20221216"]
    size = -1.0
    ppu = 12423.630576923071
-    bs_mktid = 515416577
+    bsuid = 515416577
    expiry = "2022-12-16T00:00:00+00:00"
    clears = [
     { dt = "2022-08-31T18:54:46+00:00", ppu = 12423.630576923071, accum_size = -19.0, price = 12372.75, size = 1.0, cost = 0.57, tid = "0000e1a7.630f5e5a.01.01" },
--- a/piker/brokers/ib/init.py
+++ b/piker/brokers/ib/init.py
@ -22,7 +22,7 @@ Sub-modules within break into the core functionalities:
 - ``broker.py`` part for orders / trading endpoints
 - ``feed.py`` for real-time data feed endpoints
 - ``api.py`` for the core API machinery which is ``trio``-ized
-  wrapping around `ib_async`.
+  wrapping around ``ib_insync``.
 """
 from .api import (
@ -30,52 +30,29 @@ from .api import (
 )
 from .feed import (
    open_history_client,
    open_symbol_search,
    stream_quotes,
 )
 from .broker import (
-    open_trade_dialog,
+    trades_dialogue,
 )
 from .ledger import (
    norm_trade,
    norm_trade_records,
    tx_sort,
 )
 from .symbols import (
    get_mkt_info,
    open_symbol_search,
    _search_conf,
 )
 __all__ = [
    'get_client',
-    'get_mkt_info',
+    'trades_dialogue',
    'norm_trade',
    'norm_trade_records',
    'open_trade_dialog',
    'open_history_client',
    'open_symbol_search',
    'stream_quotes',
    '_search_conf',
    'tx_sort',
 ]
 _brokerd_mods: list[str] = [
    'api',
    'broker',
 ]
 _datad_mods: list[str] = [
    'feed',
    'symbols',
 ]
 # tractor RPC enable arg
-__enable_modules__: list[str] = (
+__enable_modules__: list[str] = [
-    _brokerd_mods
+    'api',
-    +
+    'feed',
-    _datad_mods
+    'broker',
-)
+]
 # passed to ``tractor.ActorNursery.start_actor()``
 _spawn_kwargs = {
@ -86,8 +63,3 @@ _spawn_kwargs = {
 # know if ``brokerd`` should be spawned with
 # ``tractor``'s aio mode.
 _infect_asyncio: bool = True
 # XXX NOTE: for now we disable symcache with this backend since
 # there is no clearly simple nor practical way to download "all
 # symbology info" for all supported venues..
 _no_symcache: bool = True
--- a/piker/brokers/ib/_flex_reports.py
+++ b/piker/brokers/ib/_flex_reports.py
@ -1,194 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 "FLEX" report processing utils.
 """
 from bidict import bidict
 import pendulum
 from pprint import pformat
 from typing import Any
 from .api import (
    get_config,
    log,
 )
 from piker.accounting import (
    open_trade_ledger,
 )
 def parse_flex_dt(
    record: str,
 ) -> pendulum.datetime:
    '''
    Parse stupid flex record datetime stamps for the `dateTime` field..
    '''
    date, ts = record.split(';')
    dt = pendulum.parse(date)
    ts = f'{ts[:2]}:{ts[2:4]}:{ts[4:]}'
    tsdt = pendulum.parse(ts)
    return dt.set(hour=tsdt.hour, minute=tsdt.minute, second=tsdt.second)
 def flex_records_to_ledger_entries(
    accounts: bidict,
    trade_entries: list[object],
 ) -> dict:
    '''
    Convert flex report entry objects into ``dict`` form, pretty much
    straight up without modification except add a `pydatetime` field
    from the parsed timestamp.
    '''
    trades_by_account = {}
    for t in trade_entries:
        entry = t.__dict__
        # XXX: LOL apparently ``toml`` has a bug
        # where a section key error will show up in the write
        # if you leave a table key as an `int`? So i guess
        # cast to strs for all keys..
        # oddly for some so-called "BookTrade" entries
        # this field seems to be blank, no cuckin clue.
        # trade['ibExecID']
        tid = str(entry.get('ibExecID') or entry['tradeID'])
        # date = str(entry['tradeDate'])
        # XXX: is it going to cause problems if a account name
        # get's lost? The user should be able to find it based
        # on the actual exec history right?
        acctid = accounts[str(entry['accountId'])]
        # probably a flex record with a wonky non-std timestamp..
        dt = entry['pydatetime'] = parse_flex_dt(entry['dateTime'])
        entry['datetime'] = str(dt)
        if not tid:
            # this is likely some kind of internal adjustment
            # transaction, likely one of the following:
            # - an expiry event that will show a "book trade" indicating
            #   some adjustment to cash balances: zeroing or itm settle.
            # - a manual cash balance position adjustment likely done by
            #   the user from the accounts window in TWS where they can
            #   manually set the avg price and size:
            #   https://api.ibkr.com/lib/cstools/faq/web1/index.html#/tag/DTWS_ADJ_AVG_COST
            log.warning(f'Skipping ID-less ledger entry:\n{pformat(entry)}')
            continue
        trades_by_account.setdefault(
            acctid, {}
        )[tid] = entry
    for acctid in trades_by_account:
        trades_by_account[acctid] = dict(sorted(
            trades_by_account[acctid].items(),
            key=lambda entry: entry[1]['pydatetime'],
        ))
    return trades_by_account
 def load_flex_trades(
    path: str | None = None,
 ) -> dict[str, Any]:
    from ib_async import flexreport, util
    conf = get_config()
    if not path:
        # load ``brokers.toml`` and try to get the flex
        # token and query id that must be previously defined
        # by the user.
        token = conf.get('flex_token')
        if not token:
            raise ValueError(
                'You must specify a ``flex_token`` field in your'
                '`brokers.toml` in order load your trade log, see our'
                'intructions for how to set this up here:\n'
                'PUT LINK HERE!'
            )
        qid = conf['flex_trades_query_id']
        # TODO: hack this into our logging
        # system like we do with the API client..
        util.logToConsole()
        # TODO: rewrite the query part of this with async..httpx?
        report = flexreport.FlexReport(
            token=token,
            queryId=qid,
        )
    else:
        # XXX: another project we could potentially look at,
        # https://pypi.org/project/ibflex/
        report = flexreport.FlexReport(path=path)
    trade_entries = report.extract('Trade')
    ln = len(trade_entries)
    log.info(f'Loaded {ln} trades from flex query')
    trades_by_account = flex_records_to_ledger_entries(
        conf['accounts'].inverse,  # reverse map to user account names
        trade_entries,
    )
    ledger_dict: dict|None
    for acctid in trades_by_account:
        trades_by_id = trades_by_account[acctid]
        with open_trade_ledger(
            'ib',
            acctid,
            allow_from_sync_code=True,
        ) as ledger_dict:
            tid_delta = set(trades_by_id) - set(ledger_dict)
            log.info(
                'New trades detected\n'
                f'{pformat(tid_delta)}'
            )
            if tid_delta:
                sorted_delta = dict(sorted(
                    {tid: trades_by_id[tid] for tid in tid_delta}.items(),
                    key=lambda entry: entry[1].pop('pydatetime'),
                ))
                ledger_dict.update(sorted_delta)
    return ledger_dict
 if __name__ == '__main__':
    import sys
    import os
    args = sys.argv
    if len(args) > 1:
        args = args[1:]
        for arg in args:
            path = os.path.abspath(arg)
            load_flex_trades(path=path)
    else:
        # expect brokers.toml to have an entry and
        # pull from the web service.
        load_flex_trades()
--- a/piker/brokers/ib/_util.py
+++ b/piker/brokers/ib/_util.py
@ -1,424 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 ``ib`` utilities and hacks suitable for use in the backend and/or as
 runnable script-programs.
 '''
 from __future__ import annotations
 import asyncio
 from datetime import (  # noqa
    datetime,
    date,
    tzinfo as TzInfo,
 )
 from functools import partial
 from typing import (
    Literal,
    TYPE_CHECKING,
 )
 import subprocess
 import tractor
 from piker.log import get_logger
 if TYPE_CHECKING:
    from .api import Client
    import i3ipc
 log = get_logger(name=__name__)
 _reset_tech: Literal[
    'vnc',
    'i3ipc_xdotool',
    # TODO: in theory we can use a different linux DE API or
    # some other type of similar window scanning/mgmt client
    # (on other OSs) to do the same.
 ] = 'vnc'
 no_setup_msg:str = (
    'No data reset hack test setup for {vnc_sockaddr}!\n'
    'See config setup tips @\n'
    'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
 )
 def try_xdo_manual(
    client: Client,
 ):
    '''
    Do the "manual" `xdo`-based screen switch + click
    combo since apparently the `asyncvnc` client ain't workin..
    Note this is only meant as a backup method for Xorg users,
    ideally you can use a real vnc client and the `vnc_click_hack()`
    impl!
    '''
    global _reset_tech
    try:
        i3ipc_xdotool_manual_click_hack()
        _reset_tech = 'i3ipc_xdotool'
        return True
    except OSError:
        vnc_sockaddr: str = client.conf.vnc_addrs
        log.exception(
            no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
        )
        return False
 async def data_reset_hack(
    client: Client,
    reset_type: Literal['data', 'connection'],
 ) -> None:
    '''
    Run key combos for resetting data feeds and yield back to caller
    when complete.
    NOTE: this is a linux-only hack around!
    There are multiple "techs" you can use depending on your infra setup:
    - if running ib-gw in a container with a VNC server running the most
      performant method is the `'vnc'` option.
    - if running ib-gw/tws locally, and you are using `i3` you can use
      the ``i3ipc`` lib and ``xdotool`` to send the appropriate click
      and key-combos automatically to your local desktop's java X-apps.
    https://interactivebrokers.github.io/tws-api/historical_limitations.html#pacing_violations
    TODOs:
        - a return type that hopefully determines if the hack was
          successful.
        - other OS support?
        - integration with ``ib-gw`` run in docker + Xorg?
        - is it possible to offer a local server that can be accessed by
          a client? Would be sure be handy for running native java blobs
          that need to be wrangle.
    '''
    # look up any user defined vnc socket address mapped from
    # a particular API socket port.
    vnc_addrs: tuple[str]|None = client.conf.get('vnc_addrs')
    if not vnc_addrs:
        log.warning(
            no_setup_msg.format(vnc_sockaddr=client.conf)
            +
            'REQUIRES A `vnc_addrs: array` ENTRY'
        )
    global _reset_tech
    match _reset_tech:
        case 'vnc':
            try:
                await tractor.to_asyncio.run_task(
                    partial(
                        vnc_click_hack,
                        client=client,
                    )
                )
            except (
                OSError,  # no VNC server avail..
                PermissionError,  # asyncvnc pw fail..
            ) as _vnc_err:
                vnc_err = _vnc_err
                try:
                    import i3ipc  # noqa  (since a deps dynamic check)
                except ModuleNotFoundError:
                    log.warning(
                        no_setup_msg.format(vnc_sockaddr=client.conf)
                    )
                    return False
                # XXX, Xorg only workaround..
                # TODO? remove now that we have `pyvnc`?
                # if vnc_host not in {
                #     'localhost',
                #     '127.0.0.1',
                # }:
                #     focussed, matches = i3ipc_fin_wins_titled()
                #     if not matches:
                #         log.warning(
                #             no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
                #         )
                #         return False
                #     else:
                #         try_xdo_manual(vnc_sockaddr)
                # localhost but no vnc-client or it borked..
                else:
                    log.error(
                        'VNC CLICK HACK FAILE with,\n'
                        f'{vnc_err!r}\n'
                    )
                    # breakpoint()
                    # try_xdo_manual(client)
        case 'i3ipc_xdotool':
            try_xdo_manual(client)
            # i3ipc_xdotool_manual_click_hack()
        case _ as tech:
            raise RuntimeError(
                f'{tech!r} is not supported for reset tech!?'
            )
    # we don't really need the ``xdotool`` approach any more B)
    return True
 async def vnc_click_hack(
    client: Client,
    reset_type: str = 'data',
    pw: str|None = None,
 ) -> None:
    '''
    Reset the data or network connection for the VNC attached
    ib-gateway using a (magic) keybinding combo.
    A vnc-server password can be set either by an input `pw` param or
    set in the client's config with the latter loaded from the user's
    `brokers.toml` in a vnc-addrs-port-mapping section,
    .. code:: toml
        [ib.vnc_addrs]
        4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
    '''
    api_port: str = str(client.ib.client.port)
    conf: dict = client.conf
    vnc_addrs: dict[int, tuple] = conf.get('vnc_addrs')
    if not vnc_addrs:
        return None
    addr_entry: dict|tuple = vnc_addrs.get(
        api_port,
        ('localhost', 5900)  # a typical default
    )
    if pw is None:
        match addr_entry:
            case (
                host,
                port,
            ):
                pass
            case {
                'host': host,
                'port': port,
                'pw': pw
            }:
                pass
            case _:
                raise ValueError(
                    f'Invalid `ib.vnc_addrs` entry ?\n'
                    f'{addr_entry!r}\n'
                )
    try:
        from pyvnc import (
            AsyncVNCClient,
            VNCConfig,
            Point,
            MOUSE_BUTTON_LEFT,
        )
    except ModuleNotFoundError:
        log.warning(
            "In order to leverage `piker`'s built-in data reset hacks, install "
            "the `pyvnc` project: https://github.com/regulad/pyvnc.git"
        )
        return
    # two different hot keys which trigger diff types of reset
    # requests B)
    key = {
        'data': 'f',
        'connection': 'r'
    }[reset_type]
    with tractor.devx.open_crash_handler(
        ignore={TimeoutError,},
    ):
        client = await AsyncVNCClient.connect(
            VNCConfig(
                host=host,
                port=port,
                password=pw,
            )
        )
        async with client:
            # move to middle of screen
            # 640x1800
            await client.move(
                Point(
                    500,  # x from left
                    400,  # y from top
                )
            )
            # in case a prior dialog win is open/active.
            await client.press('ISO_Enter')
            # ensure the ib-gw window is active
            await client.click(MOUSE_BUTTON_LEFT)
            # send the hotkeys combo B)
            await client.press(
                'Ctrl',
                'Alt',
                key,
            )  # NOTE, keys are stacked
            # XXX, sometimes a dialog asking if you want to "simulate
            # a reset" will show, in which case we want to select
            # "Yes" (by tabbing) and then hit enter.
            iters: int = 1
            delay: float = 0.3
            await asyncio.sleep(delay)
            for i in range(iters):
                log.info(f'Sending TAB {i}')
                await client.press('Tab')
                await asyncio.sleep(delay)
            for i in range(iters):
                log.info(f'Sending ENTER {i}')
                await client.press('KP_Enter')
                await asyncio.sleep(delay)
 def i3ipc_fin_wins_titled(
    titles: list[str] = [
        'Interactive Brokers',  # tws running in i3
        'IB Gateway',  # gw running in i3
        # 'IB',  # gw running in i3 (newer version?)
        # !TODO, remote vnc instance
        # -[ ] something in title (or other Con-props) that indicates
        #   this is explicitly for ibrk sw?
        #  |_[ ]  !can use modden spawn eventually!
        'TigerVNC',
        # 'vncviewer',  # the terminal..
    ],
 ) -> tuple[
    i3ipc.Con,  # orig focussed win
    list[tuple[str, i3ipc.Con]],  # matching wins by title
 ]:
    '''
    Attempt to find a local-DE window titled with an entry in
    `titles`.
    If found deliver the current focussed window and all matching
    `i3ipc.Con`s in a list.
    '''
    import i3ipc
    ipc = i3ipc.Connection()
    # TODO: might be worth offering some kinda api for grabbing
    # the window id from the pid?
    # https://stackoverflow.com/a/2250879
    tree = ipc.get_tree()
    focussed: i3ipc.Con = tree.find_focused()
    matches: list[i3ipc.Con] = []
    for name in titles:
        results = tree.find_titled(name)
        print(f'results for {name}: {results}')
        if results:
            con = results[0]
            matches.append((
                name,
                con,
            ))
    return (
        focussed,
        matches,
    )
 def i3ipc_xdotool_manual_click_hack() -> None:
    '''
    Do the data reset hack but expecting a local X-window using `xdotool`.
    '''
    focussed, matches = i3ipc_fin_wins_titled()
    try:
        orig_win_id = focussed.window
    except AttributeError:
        # XXX if .window cucks we prolly aren't intending to
        # use this and/or just woke up from suspend..
        log.exception('xdotool invalid usage ya ??\n')
        return
    try:
        for name, con in matches:
            print(f'Resetting data feed for {name}')
            win_id = str(con.window)
            w, h = con.rect.width, con.rect.height
            # TODO: seems to be a few libs for python but not sure
            # if they support all the sub commands we need, order of
            # most recent commit history:
            # https://github.com/rr-/pyxdotool
            # https://github.com/ShaneHutter/pyxdotool
            # https://github.com/cphyc/pyxdotool
            # TODO: only run the reconnect (2nd) kc on a detected
            # disconnect?
            for key_combo, timeout in [
                # only required if we need a connection reset.
                # ('ctrl+alt+r', 12),
                # data feed reset.
                ('ctrl+alt+f', 6)
            ]:
                subprocess.call([
                    'xdotool',
                    'windowactivate', '--sync', win_id,
                    # move mouse to bottom left of window (where
                    # there should be nothing to click).
                    'mousemove_relative', '--sync', str(w-4), str(h-4),
                    # NOTE: we may need to stick a `--retry 3` in here..
                    'click', '--window', win_id,
                    '--repeat', '3', '1',
                    # hackzorzes
                    'key', key_combo,
                    ],
                    timeout=timeout,
                )
        # re-activate and focus original window
        subprocess.call([
            'xdotool',
            'windowactivate', '--sync', str(orig_win_id),
            'click', '--window', str(orig_win_id), '1',
        ])
    except subprocess.TimeoutExpired:
        log.exception('xdotool timed out?')
--- a/piker/brokers/ib/api.py
+++ b/piker/brokers/ib/api.py
--- a/piker/brokers/ib/broker.py
+++ b/piker/brokers/ib/broker.py
--- a/piker/brokers/ib/feed.py
+++ b/piker/brokers/ib/feed.py
--- a/piker/brokers/ib/ledger.py
+++ b/piker/brokers/ib/ledger.py
@ -1,532 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Trade transaction accounting and normalization.
 '''
 from __future__ import annotations
 from bisect import insort
 from dataclasses import asdict
 from decimal import Decimal
 from functools import partial
 from pprint import pformat
 from typing import (
    Any,
    Callable,
    TYPE_CHECKING,
 )
 from bidict import bidict
 from pendulum import (
    DateTime,
    parse,
    from_timestamp,
 )
 from ib_async import (
    Contract,
    Commodity,
    Fill,
    Execution,
    CommissionReport,
 )
 from piker.log import get_logger
 from piker.types import Struct
 from piker.data import (
    SymbologyCache,
 )
 from piker.accounting import (
    Asset,
    dec_digits,
    digits_to_dec,
    Transaction,
    MktPair,
    iter_by_dt,
 )
 from ._flex_reports import parse_flex_dt
 if TYPE_CHECKING:
    from .api import (
        Client,
        MethodProxy,
    )
 log = get_logger(
    name=__name__,
 )
 tx_sort: Callable = partial(
    iter_by_dt,
    parsers={
        'dateTime': parse_flex_dt,
        'datetime': parse,
        # XXX: for some some fucking 2022 and
        # back options records.. f@#$ me..
        'date': parse,
    }
 )
 def norm_trade(
    tid: str,
    record: dict[str, Any],
    # this is the dict that was returned from
    # `Client.get_mkt_pairs()` and when running offline ledger
    # processing from `.accounting`, this will be the table loaded
    # into `SymbologyCache.pairs`.
    pairs: dict[str, Struct],
    symcache: SymbologyCache | None = None,
 ) -> Transaction | None:
    conid: int = str(record.get('conId') or record['conid'])
    bs_mktid: str = str(conid)
    # NOTE: sometimes weird records (like BTTX?)
    # have no field for this?
    comms: float = -1 * (
        record.get('commission')
        or record.get('ibCommission')
        or 0
    )
    if not comms:
        log.warning(
            'No commissions found for record?\n'
            f'{pformat(record)}\n'
        )
    price: float = (
        record.get('price')
        or record.get('tradePrice')
    )
    if price is None:
        log.warning(
            'No `price` field found in record?\n'
            'Skipping normalization..\n'
            f'{pformat(record)}\n'
        )
        return None
    # the api doesn't do the -/+ on the quantity for you but flex
    # records do.. are you fucking serious ib...!?
    size: float|int = (
        record.get('quantity')
        or record['shares']
    ) * {
        'BOT': 1,
        'SLD': -1,
    }[record['side']]
    symbol: str = record['symbol']
    exch: str = (
        record.get('listingExchange')
        or record.get('primaryExchange')
        or record['exchange']
    )
    # NOTE: remove null values since `tomlkit` can't serialize
    # them to file.
    if dnc := record.pop('deltaNeutralContract', None):
        record['deltaNeutralContract'] = dnc
    # likely an opts contract record from a flex report..
    # TODO: no idea how to parse ^ the strike part from flex..
    # (00010000 any, or 00007500 tsla, ..)
    # we probably must do the contract lookup for this?
    if (
        '   ' in symbol
        or '--' in exch
    ):
        underlying, _, tail = symbol.partition('   ')
        exch: str = 'opt'
        expiry: str = tail[:6]
        # otype = tail[6]
        # strike = tail[7:]
        log.warning(
            f'Skipping option contract -> NO SUPPORT YET!\n'
            f'{symbol}\n'
        )
        return None
    # timestamping is way different in API records
    dtstr: str = record.get('datetime')
    date: str = record.get('date')
    flex_dtstr: str = record.get('dateTime')
    if dtstr or date:
        dt: DateTime = parse(dtstr or date)
    elif flex_dtstr:
        # probably a flex record with a wonky non-std timestamp..
        dt: DateTime = parse_flex_dt(record['dateTime'])
    # special handling of symbol extraction from
    # flex records using some ad-hoc schema parsing.
    asset_type: str = (
        record.get('assetCategory')
        or record.get('secType')
        or 'STK'
    )
    if (expiry := (
            record.get('lastTradeDateOrContractMonth')
            or record.get('expiry')
        )
    ):
        expiry: str = str(expiry).strip(' ')
        # NOTE: we directly use the (simple and usually short)
        # date-string expiry token when packing the `MktPair`
        # since we want the fqme to contain *that* token.
        # It might make sense later to instead parse and then
        # render different output str format(s) for this same
        # purpose depending on asset-type-market down the road.
        # Eg. for derivs we use the short token only for fqme
        # but use the isoformat('T') for transactions and
        # account file position entries?
        # dt_str: str = pendulum.parse(expiry).isoformat('T')
    # XXX: pretty much all legacy market assets have a fiat
    # currency (denomination) determined by their venue.
    currency: str = record['currency']
    src = Asset(
        name=currency.lower(),
        atype='fiat',
        tx_tick=Decimal('0.01'),
    )
    match asset_type:
        case 'FUT':
            # XXX (flex) ledger entries don't necessarily have any
            # simple 3-char key.. sometimes the .symbol is some
            # weird internal key that we probably don't want in the
            # .fqme => we should probably just wrap `Contract` to
            # this like we do other crypto$ backends XD
            # NOTE: at least older FLEX records should have
            # this field.. no idea about API entries..
            local_symbol: str | None = record.get('localSymbol')
            underlying_key: str = record.get('underlyingSymbol')
            descr: str | None = record.get('description')
            if (
                not (
                    local_symbol
                    and symbol in local_symbol
                )
                and (
                        descr
                        and symbol not in descr
                    )
            ):
                con_key, exp_str = descr.split(' ')
                symbol: str = underlying_key or con_key
            dst = Asset(
                name=symbol.lower(),
                atype='future',
                tx_tick=Decimal('1'),
            )
        case 'STK':
            dst = Asset(
                name=symbol.lower(),
                atype='stock',
                tx_tick=Decimal('1'),
            )
        case 'CASH':
            if currency not in symbol:
                # likely a dict-casted `Forex` contract which
                # has .symbol as the dst and .currency as the
                # src.
                name: str = symbol.lower()
            else:
                # likely a flex-report record which puts
                # EUR.USD as the symbol field and just USD in
                # the currency field.
                name: str = symbol.lower().replace(f'.{src.name}', '')
            dst = Asset(
                name=name,
                atype='fiat',
                tx_tick=Decimal('0.01'),
            )
        case 'OPT':
            dst = Asset(
                name=symbol.lower(),
                atype='option',
                tx_tick=Decimal('1'),
                # TODO: we should probably always cast to the
                # `Contract` instance then dict-serialize that for
                # the `.info` field!
                # info=asdict(Option()),
            )
        case 'CMDTY':
            from .symbols import _adhoc_symbol_map
            con_kwargs, _ = _adhoc_symbol_map[symbol.upper()]
            dst = Asset(
                name=symbol.lower(),
                atype='commodity',
                tx_tick=Decimal('1'),
                info=asdict(Commodity(**con_kwargs)),
            )
    # try to build out piker fqme from record.
    # src: str = record['currency']
    price_tick: Decimal = digits_to_dec(dec_digits(price))
    # NOTE: can't serlialize `tomlkit.String` so cast to native
    atype: str = str(dst.atype)
    # if not (mkt := symcache.mktmaps.get(bs_mktid)):
    mkt = MktPair(
        bs_mktid=bs_mktid,
        dst=dst,
        price_tick=price_tick,
        # NOTE: for "legacy" assets, volume is normally discreet, not
        # a float, but we keep a digit in case the suitz decide
        # to get crazy and change it; we'll be kinda ready
        # schema-wise..
        size_tick=Decimal('1'),
        src=src,  # XXX: normally always a fiat
        _atype=atype,
        venue=exch,
        expiry=expiry,
        broker='ib',
        _fqme_without_src=(atype != 'fiat'),
    )
    fqme: str = mkt.fqme
    # XXX: if passed in, we fill out the symcache ad-hoc in order
    # to make downstream accounting work..
    if symcache is not None:
        orig_mkt: MktPair | None  = symcache.mktmaps.get(bs_mktid)
        if (
            orig_mkt
            and orig_mkt.fqme != mkt.fqme
        ):
            log.warning(
            # print(
                f'Contracts with common `conId`: {bs_mktid} mismatch..\n'
                f'{orig_mkt.fqme} -> {mkt.fqme}\n'
                # 'with DIFF:\n'
                # f'{mkt - orig_mkt}'
            )
        symcache.mktmaps[bs_mktid] = mkt
        symcache.mktmaps[fqme] = mkt
        symcache.assets[src.name] = src
        symcache.assets[dst.name] = dst
    # NOTE: for flex records the normal fields for defining an fqme
    # sometimes won't be available so we rely on two approaches for
    # the "reverse lookup" of piker style fqme keys:
    # - when dealing with API trade records received from
    #   `IB.trades()` we do a contract lookup at he time of processing
    # - when dealing with flex records, it is assumed the record
    #   is at least a day old and thus the TWS position reporting system
    #   should already have entries if the pps are still open, in
    #   which case, we can pull the fqme from that table (see
    #   `trades_dialogue()` above).
    return Transaction(
        fqme=fqme,
        tid=tid,
        size=size,
        price=price,
        cost=comms,
        dt=dt,
        expiry=expiry,
        bs_mktid=str(conid),
    )
 def norm_trade_records(
    ledger: dict[str, Any],
    symcache: SymbologyCache | None = None,
 ) -> dict[str, Transaction]:
    '''
    Normalize (xml) flex-report or (recent) API trade records into
    our ledger format with parsing for `MktPair` and `Asset`
    extraction to fill in the `Transaction.sys: MktPair` field.
    '''
    records: list[Transaction] = []
    for tid, record in ledger.items():
        txn = norm_trade(
            tid,
            record,
            # NOTE: currently no symcache support
            pairs={},
            symcache=symcache,
        )
        if txn is None:
            continue
        # inject txns sorted by datetime
        insort(
            records,
            txn,
            key=lambda t: t.dt
        )
    return {r.tid: r for r in records}
 def api_trades_to_ledger_entries(
    accounts: bidict[str, str],
    fills: list[Fill],
 ) -> dict[str, dict]:
    '''
    Convert API execution objects entry objects into
    flattened-``dict`` form, pretty much straight up without
    modification except add a `pydatetime` field from the parsed
    timestamp so that on write
    '''
    trades_by_account: dict[str, dict] = {}
    for fill in fills:
        # NOTE: for the schema, see the defn for `Fill` which is
        # a `NamedTuple` subtype
        fdict: dict = fill._asdict()
        # flatten all (sub-)objects and convert to dicts.
        # with values packed into one top level entry.
        val: CommissionReport | Execution | Contract
        txn_dict: dict[str, Any] = {}
        for attr_name, val in fdict.items():
            match attr_name:
                # value is a `@dataclass` subtype
                case 'contract' | 'execution' | 'commissionReport':
                    txn_dict.update(asdict(val))
                case 'time':
                    # ib has wack ns timestamps, or is that us?
                    continue
                # TODO: we can remove this case right since there's
                # only 4 fields on a `Fill`?
                case _:
                    txn_dict[attr_name] = val
        tid = str(txn_dict['execId'])
        dt = from_timestamp(txn_dict['time'])
        txn_dict['datetime'] = str(dt)
        acctid = accounts[txn_dict['acctNumber']]
        # NOTE: only inserted (then later popped) for sorting below!
        txn_dict['pydatetime'] = dt
        if not tid:
            # this is likely some kind of internal adjustment
            # transaction, likely one of the following:
            # - an expiry event that will show a "book trade" indicating
            #   some adjustment to cash balances: zeroing or itm settle.
            # - a manual cash balance position adjustment likely done by
            #   the user from the accounts window in TWS where they can
            #   manually set the avg price and size:
            #   https://api.ibkr.com/lib/cstools/faq/web1/index.html#/tag/DTWS_ADJ_AVG_COST
            log.warning(
                'Skipping ID-less ledger txn_dict:\n'
                f'{pformat(txn_dict)}'
            )
            continue
        trades_by_account.setdefault(
            acctid, {}
        )[tid] = txn_dict
    # TODO: maybe we should just bisect.insort() into a list of
    # tuples and then return a dict of that?
    # sort entries in output by python based datetime
    for acctid in trades_by_account:
        trades_by_account[acctid] = dict(sorted(
            trades_by_account[acctid].items(),
            key=lambda entry: entry[1].pop('pydatetime'),
        ))
    return trades_by_account
 async def update_ledger_from_api_trades(
    fills: list[Fill],
    client: Client | MethodProxy,
    accounts_def_inv: bidict[str, str],
    # NOTE: provided for ad-hoc insertions "as transactions are
    # processed" -> see `norm_trade()` signature requirements.
    symcache: SymbologyCache | None = None,
 ) -> tuple[
    dict[str, Transaction],
    dict[str, dict],
 ]:
    # XXX; ERRGGG..
    # pack in the "primary/listing exchange" value from a
    # contract lookup since it seems this isn't available by
    # default from the `.fills()` method endpoint...
    fill: Fill
    for fill in fills:
        con: Contract = fill.contract
        conid: str = con.conId
        pexch: str | None = con.primaryExchange
        if not pexch:
            cons = await client.get_con(conid=conid)
            if cons:
                con = cons[0]
                pexch = con.primaryExchange or con.exchange
            else:
                # for futes it seems like the primary is always empty?
                pexch: str = con.exchange
        # pack in the ``Contract.secType``
        # entry['asset_type'] = condict['secType']
    entries: dict[str, dict] = api_trades_to_ledger_entries(
        accounts_def_inv,
        fills,
    )
    # normalize recent session's trades to the `Transaction` type
    trans_by_acct: dict[str, dict[str, Transaction]] = {}
    for acctid, trades_by_id in entries.items():
        # normalize to transaction form
        trans_by_acct[acctid] = norm_trade_records(
            trades_by_id,
            symcache=symcache,
        )
    return trans_by_acct, entries
--- a/piker/brokers/ib/symbols.py
+++ b/piker/brokers/ib/symbols.py
@ -1,650 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Symbology search and normalization.
 '''
 from __future__ import annotations
 from contextlib import (
    nullcontext,
 )
 from decimal import Decimal
 from functools import partial
 import time
 from typing import (
    Awaitable,
    TYPE_CHECKING,
 )
 from rapidfuzz import process as fuzzy
 import ib_async as ibis
 import tractor
 from tractor.devx.pformat import ppfmt
 import trio
 from piker.accounting import (
    Asset,
    MktPair,
    unpack_fqme,
 )
 from piker._cacheables import (
    async_lifo_cache,
 )
 from piker.log import get_logger
 if TYPE_CHECKING:
    from .api import (
        MethodProxy,
        Client,
    )
 log = get_logger(
    name=__name__,
 )
 _futes_venues = (
    'GLOBEX',
    'NYMEX',
    'CME',
    'CMECRYPTO',
    'COMEX',
    # 'CMDTY',  # special name case..
    'CBOT',  # (treasury) yield futures
 )
 _adhoc_cmdty_set = {
    # metals
    # https://misc.interactivebrokers.com/cstools/contract_info/v3.10/index.php?action=Conid%20Info&wlId=IB&conid=69067924
    'xauusd.cmdty',  # london gold spot ^
    'xagusd.cmdty',  # silver spot
 }
 # NOTE: if you aren't seeing one of these symbol's futues contracts
 # show up, it's likely the `.<venue>` part is wrong!
 _adhoc_futes_set = {
    # equities
    'nq.cme',
    'mnq.cme',  # micro
    'es.cme',
    'mes.cme',  # micro
    # cypto$
    'brr.cme',
    'mbt.cme',  # micro
    'ethusdrr.cme',
    # agriculture
    'he.comex',  # lean hogs
    'le.comex',  # live cattle (geezers)
    'gf.comex',  # feeder cattle (younguns)
    # raw
    'lb.comex',  # random len lumber
    'gc.comex',
    'mgc.comex',  # micro
    # oil & gas
    'cl.nymex',
    'ni.comex',  # silver futes
    'qi.comex',  # mini-silver futes
    # treasury yields
    # etfs by duration:
    # SHY -> IEI -> IEF -> TLT
    'zt.cbot',  # 2y
    'z3n.cbot',  # 3y
    'zf.cbot',  # 5y
    'zn.cbot',  # 10y
    'zb.cbot',  # 30y
    # (micros of above)
    '2yy.cbot',
    '5yy.cbot',
    '10y.cbot',
    '30y.cbot',
 }
 # taken from list here:
 # https://www.interactivebrokers.com/en/trading/products-spot-currencies.php
 _adhoc_fiat_set = set((
    'USD, AED, AUD, CAD,'
    'CHF, CNH, CZK, DKK,'
    'EUR, GBP, HKD, HUF,'
    'ILS, JPY, MXN, NOK,'
    'NZD, PLN, RUB, SAR,'
    'SEK, SGD, TRY, ZAR'
    ).split(' ,')
 )
 # manually discovered tick discrepancies,
 # onl god knows how or why they'd cuck these up..
 _adhoc_mkt_infos: dict[int|str, dict] = {
    'vtgn.nasdaq': {'price_tick': Decimal('0.01')},
 }
 # map of symbols to contract ids
 _adhoc_symbol_map = {
    # https://misc.interactivebrokers.com/cstools/contract_info/v3.10/index.php?action=Conid%20Info&wlId=IB&conid=69067924
    # NOTE: some cmdtys/metals don't have trade data like gold/usd:
    # https://groups.io/g/twsapi/message/44174
    'XAUUSD': ({'conId': 69067924}, {'whatToShow': 'MIDPOINT'}),
 }
 for qsn in _adhoc_futes_set:
    sym, venue = qsn.split('.')
    assert venue.upper() in _futes_venues, f'{venue}'
    _adhoc_symbol_map[sym.upper()] = (
        {'exchange': venue},
        {},
    )
 # exchanges we don't support at the moment due to not knowing
 # how to do symbol-contract lookup correctly likely due
 # to not having the data feeds subscribed.
 _exch_skip_list = {
    'ASX',  # aussie stocks
    'MEXI',  # mexican stocks
    # no idea
    'NSE',
    'VALUE',
    'FUNDSERV',
    'SWB2',
    'PSE',
    'PHLX',
 }
 # optional search config the backend can register for
 # it's symbol search handling (in this case we avoid
 # accepting patterns before the kb has settled more then
 # a quarter second).
 _search_conf = {
    'pause_period': 6 / 16,
 }
@tractor.context
 async def open_symbol_search(ctx: tractor.Context) -> None:
    '''
    Symbology search brokerd-endpoint.
    '''
    from .api import open_client_proxies
    from .feed import open_data_client
    # TODO: load user defined symbol set locally for fast search?
    await ctx.started({})
    async with (
        open_client_proxies() as (proxies, _),
        open_data_client() as data_proxy,
    ):
        async with ctx.open_stream() as stream:
            # select a non-history client for symbol search to lighten
            # the load in the main data node.
            proxy = data_proxy
            for name, proxy in proxies.items():
                if proxy is data_proxy:
                    continue
                break
            ib_client = proxy._aio_ns.ib
            log.info(
                f'Using API client for symbol-search\n'
                f'{ib_client}\n'
            )
            last: float = time.time()
            async for pattern in stream:
                log.info(f'received {pattern}')
                now: float = time.time()
                # TODO? check this is no longer true?
                # this causes tractor hang...
                # assert 0
                assert pattern, 'IB can not accept blank search pattern'
                # throttle search requests to no faster then 1Hz
                diff: float = now - last
                if diff < 1.0:
                    log.debug('throttle sleeping')
                    await trio.sleep(diff)
                    try:
                        pattern = stream.receive_nowait()
                    except trio.WouldBlock:
                        pass
                if (
                    not pattern
                    or
                    pattern.isspace()
                    or
                    # XXX: not sure if this is a bad assumption but it
                    # seems to make search snappier?
                    len(pattern) < 1
                ):
                    log.warning('empty pattern received, skipping..')
                    # TODO: *BUG* if nothing is returned here the client
                    # side will cache a null set result and not showing
                    # anything to the use on re-searches when this query
                    # timed out. We probably need a special "timeout" msg
                    # or something...
                    # XXX: this unblocks the far end search task which may
                    # hold up a multi-search nursery block
                    await stream.send({})
                    continue
                log.info(
                    f'Searching for FQME with,\n'
                    f'pattern: {pattern!r}\n'
                )
                last: float = time.time()
                # async batch search using api stocks endpoint and
                # module defined adhoc symbol set.
                stock_results: list[dict] = []
                async def extend_results(
                    # ?TODO, how to type async-fn!?
                    target: Awaitable[list],
                    pattern: str,
                    **kwargs,
                ) -> None:
                    try:
                        results = await target(
                            pattern=pattern,
                            **kwargs,
                        )
                        client_repr: str = proxy._aio_ns.ib.client.__class__.__name__
                        meth_repr: str = target.keywords["meth"]
                        log.info(
                            f'Search query,\n'
                            f'{client_repr}.{meth_repr}(\n'
                            f'    pattern={pattern!r}\n'
                            f'    **kwargs={kwargs!r},\n'
                            f') = {ppfmt(list(results))}'
                            # XXX ^ just the keys since that's what
                            # shows in UI results table.
                        )
                    except tractor.trionics.Lagged:
                        log.exception(
                            'IB SYM-SEARCH OVERRUN?!?\n'
                        )
                        return
                    stock_results.extend(results)
                for _ in range(10):
                    with trio.move_on_after(3) as cs:
                        async with trio.open_nursery() as tn:
                            tn.start_soon(
                                partial(
                                    extend_results,
                                    pattern=pattern,
                                    target=proxy.search_symbols,
                                    upto=10,
                                ),
                            )
                            # trigger async request
                            await trio.sleep(0)
                    if cs.cancelled_caught:
                        log.warning(
                            f'Search timeout? {proxy._aio_ns.ib.client}'
                        )
                        continue
                    elif stock_results:
                        break
                    # else:
                    # await tractor.pause()
                    # # match against our ad-hoc set immediately
                    # adhoc_matches = fuzzy.extract(
                    #     pattern,
                    #     list(_adhoc_futes_set),
                    #     score_cutoff=90,
                    # )
                    # log.info(f'fuzzy matched adhocs: {adhoc_matches}')
                    # adhoc_match_results = {}
                    # if adhoc_matches:
                    #     # TODO: do we need to pull contract details?
                    #     adhoc_match_results = {i[0]: {} for i in
                    #     adhoc_matches}
                log.debug(
                    f'fuzzy matching stocks {ppfmt(stock_results)}'
                )
                stock_matches = fuzzy.extract(
                    pattern,
                    stock_results,
                    score_cutoff=50,
                )
                # matches = adhoc_match_results | {
                matches = {
                    item[0]: {} for item in stock_matches
                }
                # TODO: we used to deliver contract details
                # {item[2]: item[0] for item in stock_matches}
                log.debug(
                    f'Sending final matches\n'
                    f'{matches.keys()}'
                )
                await stream.send(matches)
 # re-mapping to piker asset type names
 # https://github.com/erdewit/ib_insync/blob/master/ib_insync/contract.py#L113
 _asset_type_map = {
    'STK': 'stock',
    'OPT': 'option',
    'FUT': 'future',
    'CONTFUT': 'continuous_future',
    'CASH': 'fiat',
    'IND': 'index',
    'CFD': 'cfd',
    'BOND': 'bond',
    'CMDTY': 'commodity',
    'FOP': 'futures_option',
    'FUND': 'mutual_fund',
    'WAR': 'warrant',
    'IOPT': 'warran',
    'BAG': 'bag',
    'CRYPTO': 'crypto',  # bc it's diff then fiat?
    # 'NEWS': 'news',
 }
 def parse_patt2fqme(
    # client: Client,
    pattern: str,
 ) -> tuple[str, str, str, str]:
    # TODO: we can't use this currently because
    # ``wrapper.starTicker()`` currently cashes ticker instances
    # which means getting a singel quote will potentially look up
    # a quote for a ticker that it already streaming and thus run
    # into state clobbering (eg. list: Ticker.ticks). It probably
    # makes sense to try this once we get the pub-sub working on
    # individual symbols...
    # XXX UPDATE: we can probably do the tick/trades scraping
    # inside our eventkit handler instead to bypass this entirely?
    currency = ''
    # fqme parsing stage
    # ------------------
    if '.ib' in pattern:
        _, symbol, venue, expiry = unpack_fqme(pattern)
    else:
        symbol = pattern
        expiry = ''
        # # another hack for forex pairs lul.
        # if (
        #     '.idealpro' in symbol
        #     # or '/' in symbol
        # ):
        #     exch: str = 'IDEALPRO'
        #     symbol = symbol.removesuffix('.idealpro')
        #     if '/' in symbol:
        #         symbol, currency = symbol.split('/')
        # else:
        # TODO: yes, a cache..
        # try:
        #     # give the cache a go
        #     return client._contracts[symbol]
        # except KeyError:
        #     log.debug(f'Looking up contract for {symbol}')
        expiry: str = ''
        if symbol.count('.') > 1:
            symbol, _, expiry = symbol.rpartition('.')
        # use heuristics to figure out contract "type"
        symbol, venue = symbol.upper().rsplit('.', maxsplit=1)
    return symbol, currency, venue, expiry
 def con2fqme(
    con: ibis.Contract,
    _cache: dict[int, (str, bool)] = {}
 ) -> tuple[str, bool]:
    '''
    Convert contracts to fqme-style strings to be used both in
    symbol-search matching and as feed tokens passed to the front
    end data deed layer.
    Previously seen contracts are cached by id.
    '''
    # should be real volume for this contract by default
    calc_price: bool = False
    if con.conId:
        try:
            # TODO: LOL so apparently IB just changes the contract
            # ID (int) on a whim.. so we probably need to use an
            # FQME style key after all...
            return _cache[con.conId]
        except KeyError:
            pass
    suffix: str = con.primaryExchange or con.exchange
    symbol: str = con.symbol
    expiry: str = con.lastTradeDateOrContractMonth or ''
    match con:
        case ibis.Option():
            # TODO: option symbol parsing and sane display:
            symbol = con.localSymbol.replace(' ', '')
        case (
            ibis.Commodity()
            # search API endpoint returns std con box..
            | ibis.Contract(secType='CMDTY')
        ):
            # commodities and forex don't have an exchange name and
            # no real volume so we have to calculate the price
            suffix = con.secType
            # no real volume on this tract
            calc_price = True
        case ibis.Forex() | ibis.Contract(secType='CASH'):
            dst, src = con.localSymbol.split('.')
            symbol = ''.join([dst, src])
            suffix = con.exchange or 'idealpro'
            # no real volume on forex feeds..
            calc_price = True
    if not suffix:
        entry = _adhoc_symbol_map.get(
            con.symbol or con.localSymbol
        )
        if entry:
            meta, kwargs = entry
            cid = meta.get('conId')
            if cid:
                assert con.conId == meta['conId']
            suffix = meta['exchange']
    # append a `.<suffix>` to the returned symbol
    # key for derivatives that normally is the expiry
    # date key.
    if expiry:
        suffix += f'.{expiry}'
    fqme_key = symbol.lower()
    if suffix:
        fqme_key = '.'.join((fqme_key, suffix)).lower()
    _cache[con.conId] = fqme_key, calc_price
    return fqme_key, calc_price
@async_lifo_cache()
 async def get_mkt_info(
    fqme: str,
    proxy: MethodProxy|None = None,
 ) -> tuple[MktPair, ibis.ContractDetails]:
    if '.ib' not in fqme:
        fqme += '.ib'
    broker, pair, venue, expiry = unpack_fqme(fqme)
    proxy: MethodProxy
    if proxy is not None:
        client_ctx = nullcontext(proxy)
    else:
        from .feed import (
            open_data_client,
        )
        client_ctx = open_data_client
    async with client_ctx as proxy:
        try:
            (
                con,  # Contract
                details,  # ContractDetails
            ) = await proxy.get_sym_details(fqme=fqme)
        except ConnectionError:
            log.exception(f'Proxy is ded {proxy._aio_ns}')
            raise
    # TODO: more consistent field translation
    atype = _asset_type_map[con.secType]
    if atype == 'commodity':
        venue: str = 'cmdty'
    else:
        venue: str = (
            con.primaryExchange
            or
            con.exchange
        )
    price_tick: Decimal = Decimal(str(details.minTick))
    ib_min_tick_gt_2: Decimal = Decimal('0.01')
    if (
        price_tick < ib_min_tick_gt_2
    ):
        # TODO: we need to add some kinda dynamic rounding sys
        # to our MktPair i guess?
        # not sure where the logic should sit, but likely inside
        # the `.clearing._ems` i suppose...
        log.warning(
            'IB seems to disallow a min price tick < 0.01 '
            'when the price is > 2.0..?\n'
            f'Decreasing min tick precision for {fqme} to 0.01'
        )
        # price_tick = ib_min_tick
        # await tractor.pause()
    if atype == 'stock':
        # XXX: GRRRR they don't support fractional share sizes for
        # stocks from the API?!
        # if con.secType == 'STK':
        size_tick = Decimal('1')
    else:
        size_tick: Decimal = Decimal(
            str(details.minSize).rstrip('0')
        )
        # ?TODO, there is also the Contract.sizeIncrement, bt wtf is it?
    # NOTE: this is duplicate from the .broker.norm_trade_records()
    # routine, we should factor all this parsing somewhere..
    expiry_str = str(con.lastTradeDateOrContractMonth)
    # if expiry:
    #     expiry_str: str = str(pendulum.parse(
    #         str(expiry).strip(' ')
    #     ))
    # TODO: currently we can't pass the fiat src asset because
    # then we'll get a `MNQUSD` request for history data..
    # we need to figure out how we're going to handle this (later?)
    # but likely we want all backends to eventually handle
    # ``dst/src.venue.`` style !?
    src = Asset(
        name=str(con.currency).lower(),
        atype='fiat',
        tx_tick=Decimal('0.01'),  # right?
    )
    dst = Asset(
        name=con.symbol.lower(),
        atype=atype,
        tx_tick=size_tick,
    )
    mkt = MktPair(
        src=src,
        dst=dst,
        price_tick=price_tick,
        size_tick=size_tick,
        bs_mktid=str(con.conId),
        venue=str(venue),
        expiry=expiry_str,
        broker='ib',
        # TODO: options contract info as str?
        # contract_info=<optionsdetails>
        _fqme_without_src=(atype != 'fiat'),
    )
    # just.. wow.
    if entry := _adhoc_mkt_infos.get(mkt.bs_fqme):
        log.warning(f'Frickin {mkt.fqme} has an adhoc {entry}..')
        new = mkt.to_dict()
        new['price_tick'] = entry['price_tick']
        new['src'] = src
        new['dst'] = dst
        mkt = MktPair(**new)
    # if possible register the bs_mktid to the just-built
    # mkt so that it can be retreived by order mode tasks later.
    # TODO NOTE: this is going to be problematic if/when we split
    # out the datatd vs. brokerd actors since the mktmap lookup
    # table will now be inaccessible..
    if proxy is not None:
        client: Client = proxy._aio_ns
        client._contracts[mkt.bs_fqme] = con
        client._cons2mkts[con] = mkt
    return mkt, details
--- a/piker/brokers/ib/venues.py
+++ b/piker/brokers/ib/venues.py
@ -1,325 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 (Multi-)venue mgmt helpers.
 IB generally supports all "legacy" trading venues, those mostly owned
 by ICE and friends.
 '''
 from __future__ import annotations
 from datetime import (  # noqa
    datetime,
    date,
    tzinfo as TzInfo,
 )
 from typing import (
    Iterator,
    TYPE_CHECKING,
 )
 import exchange_calendars as xcals
 from pendulum import (
    now,
    Duration,
    Interval,
    Time,
 )
 if TYPE_CHECKING:
    from ib_async import (
        TradingSession,
        Contract,
        ContractDetails,
    )
    from exchange_calendars.exchange_calendars import (
        ExchangeCalendar,
    )
    from pandas import (
        # DatetimeIndex,
        TimeDelta,
        Timestamp,
    )
 def has_weekend(
    period: Interval,
 ) -> bool:
    '''
    Predicate to for a period being within
    days 6->0 (sat->sun).
    '''
    has_weekend: bool = False
    for dt in period:
        if dt.day_of_week in [0, 6]: # 0=Sunday, 6=Saturday
            has_weekend = True
            break
    return has_weekend
 def has_holiday(
    con_deats: ContractDetails,
    period: Interval,
 ) -> bool:
    '''
    Using the `exchange_calendars` lib detect if a time-gap `period`
    is contained in a known "cash hours" closure.
    '''
    tz: str = con_deats.timeZoneId
    con: Contract = con_deats.contract
    exch: str = (
        con.primaryExchange
        or
        con.exchange
    )
    # XXX, ad-hoc handle any IB exchange which are non-std
    # via lookup table..
    std_exch: dict = {
        'ARCA': 'ARCX',
    }.get(exch, exch)
    cal: ExchangeCalendar = xcals.get_calendar(std_exch)
    end: datetime = period.end
    # _start: datetime = period.start
    # ?TODO, can rm ya?
    # => not that useful?
    # dti: DatetimeIndex = cal.sessions_in_range(
    #     _start.date(),
    #     end.date(),
    # )
    prev_close: Timestamp = cal.previous_close(
        end.date()
    ).tz_convert(tz)
    prev_open: Timestamp = cal.previous_open(
        end.date()
    ).tz_convert(tz)
    # now do relative from prev_ values ^
    # to get the next open which should match
    # "contain" the end of the gap.
    next_open: Timestamp = cal.next_open(
        prev_open,
    ).tz_convert(tz)
    next_open: Timestamp = cal.next_open(
        prev_open,
    ).tz_convert(tz)
    _next_close: Timestamp = cal.next_close(
        prev_close
    ).tz_convert(tz)
    cash_gap: TimeDelta = next_open - prev_close
    is_holiday_gap = (
        cash_gap
        > 
        period
    )
    # XXX, debug
    # breakpoint()
    return is_holiday_gap
 def is_current_time_in_range(
    sesh: Interval,
    when: datetime|None = None,
 ) -> bool:
    '''
    Check if current time is within the datetime range.
    Use any/the-same timezone as provided by `start_dt.tzinfo` value
    in the range.
    '''
    when: datetime = when or now()
    return when in sesh
 def iter_sessions(
    con_deats: ContractDetails,
 ) -> Iterator[Interval]:
    '''
    Yield `pendulum.Interval`s for all
    `ibas.ContractDetails.tradingSessions() -> TradingSession`s.
    '''
    sesh: TradingSession
    for sesh in con_deats.tradingSessions():
        yield Interval(*sesh)
 def sesh_times(
    con_deats: ContractDetails,
 ) -> tuple[Time, Time]:
    '''
    Based on the earliest trading session provided by the IB API,
    get the (day-agnostic) times for the start/end.
    '''
    earliest_sesh: Interval = next(iter_sessions(con_deats))
    return (
        earliest_sesh.start.time(),
        earliest_sesh.end.time(),
    )
    # ^?TODO, use `.diff()` to get point-in-time-agnostic period?
    # https://pendulum.eustace.io/docs/#difference
 def is_venue_open(
    con_deats: ContractDetails,
    when: datetime|Duration|None = None,
 ) -> bool:
    '''
    Check if market-venue is open during `when`, which defaults to
    "now".
    '''
    sesh: Interval
    for sesh in iter_sessions(con_deats):
        if is_current_time_in_range(
            sesh=sesh,
            when=when,
        ):
            return True
    return False
 def is_venue_closure(
    gap: Interval,
    con_deats: ContractDetails,
    time_step_s: int,
 ) -> bool:
    '''
    Check if a provided time-`gap` is just an (expected) trading
    venue closure period.
    '''
    open: Time
    close: Time
    open, close = sesh_times(con_deats)
    # ensure times are in mkt-native timezone
    tz: str = con_deats.timeZoneId
    start = gap.start.in_tz(tz)
    start_t = start.time()
    end = gap.end.in_tz(tz)
    end_t = end.time()
    if (
        (
            start_t in (
                close,
                close.subtract(seconds=time_step_s)
            )
            and 
            end_t in (
                open,
                open.add(seconds=time_step_s),
            )
        )
        or
        has_weekend(gap)
        or
        has_holiday(
            con_deats=con_deats,
            period=gap,
        )
    ):
        return True
    # breakpoint()
    return False
 # TODO, put this into `._util` and call it from here!
 #
 # NOTE, this was generated by @guille from a gpt5 prompt
 # and was originally thot to be needed before learning about
 # `ib_async.contract.ContractDetails._parseSessions()` and
 # it's downstream meths..
 #
 # This is still likely useful to keep for now to parse the
 # `.tradingHours: str` value manually if we ever decide
 # to move off `ib_async` and implement our own `trio`/`anyio`
 # based version Bp
 #
 # >attempt to parse the retarted ib "time stampy thing" they
 # >do for "venue hours" with this.. written by
 # >gpt5-"thinking",
 #
 def parse_trading_hours(
    spec: str,
    tz: TzInfo|None = None
 ) -> dict[
    date,
    tuple[datetime, datetime]
 ]|None:
    '''
    Parse venue hours like:
    'YYYYMMDD:HHMM-YYYYMMDD:HHMM;YYYYMMDD:CLOSED;...'
    Returns `dict[date] = (open_dt, close_dt)` or `None` if
    closed.
    '''
    if (
        not isinstance(spec, str)
        or
        not spec
    ):
        raise ValueError('spec must be a non-empty string')
    out: dict[
        date,
        tuple[datetime, datetime]
    ]|None = {}
    for part in (p.strip() for p in spec.split(';') if p.strip()):
        if part.endswith(':CLOSED'):
            day_s, _ = part.split(':', 1)
            d = datetime.strptime(day_s, '%Y%m%d').date()
            out[d] = None
            continue
        try:
            start_s, end_s = part.split('-', 1)
            start_dt = datetime.strptime(start_s, '%Y%m%d:%H%M')
            end_dt = datetime.strptime(end_s, '%Y%m%d:%H%M')
        except ValueError as exc:
            raise ValueError(f'invalid segment: {part}') from exc
        if tz is not None:
            start_dt = start_dt.replace(tzinfo=tz)
            end_dt = end_dt.replace(tzinfo=tz)
        out[start_dt.date()] = (start_dt, end_dt)
    return out
 # ORIG desired usage,
 #
 # TODO, for non-drunk tomorrow,
 # - call above fn and check that `output[today] is not None`
 # trading_hrs: dict = parse_trading_hours(
 #     details.tradingHours
 # )
 # liq_hrs: dict = parse_trading_hours(
 #     details.liquidHours
    # )
--- a/piker/brokers/kraken/README.rst
+++ b/piker/brokers/kraken/README.rst
@ -58,7 +58,7 @@ your ``pps.toml`` file will have position entries like,
   [kraken.spot."xmreur.kraken"]
   size = 4.80907954
   ppu = 103.97000000
-   bs_mktid = "XXMRZEUR"
+   bsuid = "XXMRZEUR"
   clears = [
    { tid = "TFJBKK-SMBZS-VJ4UWS", cost = 0.8, price = 103.97, size = 4.80907954, dt = "2022-05-20T02:26:33.413397+00:00" },
   ]
--- a/piker/brokers/kraken/init.py
+++ b/piker/brokers/kraken/init.py
@ -19,57 +19,43 @@ Kraken backend.
 Sub-modules within break into the core functionalities:
- .api: for the core API machinery which generally
+- ``broker.py`` part for orders / trading endpoints
-        a ``asks``/``trio-websocket`` implemented ``Client``.
+- ``feed.py`` for real-time data feed endpoints
- .broker: part for orders / trading endpoints.
+- ``api.py`` for the core API machinery which is ``trio``-ized
- .feed: for real-time and historical data query endpoints.
+  wrapping around ``ib_insync``.
 - .ledger: for transaction processing as it pertains to accounting.
 - .symbols: for market (name) search and symbology meta-defs.
 '''
-from .symbols import (
+
-    Pair,  # for symcache
+from piker.log import get_logger
-    open_symbol_search,
+
-    # required by `.accounting`, `.data`
+log = get_logger(__name__)
-    get_mkt_info,
+
 )
 # required by `.brokers`
 from .api import (
    get_client,
 )
 from .feed import (
    # required by `.data`
    stream_quotes,
    open_history_client,
    open_symbol_search,
    stream_quotes,
 )
 from .broker import (
-    # required by `.clearing`
+    trades_dialogue,
    open_trade_dialog,
 )
 from .ledger import (
    # required by `.accounting`
    norm_trade,
    norm_trade_records,
 )
 __all__ = [
    'get_client',
-    'get_mkt_info',
+    'trades_dialogue',
    'Pair',
    'open_trade_dialog',
    'open_history_client',
    'open_symbol_search',
    'stream_quotes',
    'norm_trade_records',
    'norm_trade',
 ]
 # tractor RPC enable arg
 __enable_modules__: list[str] = [
    'api',
    'broker',
    'feed',
-    'symbols',
+    'broker',
 ]
--- a/piker/brokers/kraken/api.py
+++ b/piker/brokers/kraken/api.py
@ -15,7 +15,7 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-Core (web) API client
+Kraken web API wrapping.
 '''
 from contextlib import asynccontextmanager as acm
@ -23,52 +23,53 @@ from datetime import datetime
 import itertools
 from typing import (
    Any,
    Optional,
    Union,
 )
 import time
-import httpx
+from bidict import bidict
 import pendulum
 import asks
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 import urllib.parse
 import hashlib
 import hmac
 import base64
 import tractor
 import trio
 from piker import config
 from piker.data import (
    def_iohlcv_fields,
    match_from_pairs,
 )
 from piker.accounting._mktinfo import (
    Asset,
    digits_to_dec,
    dec_digits,
 )
 from piker.brokers._util import (
    resproc,
    SymbolNotFound,
    BrokerError,
    DataThrottle,
 )
-from piker.accounting import Transaction
+from piker.pp import Transaction
-from piker.log import get_logger
+from . import log
 from .symbols import Pair
 log = get_logger('piker.brokers.kraken')
 # <uri>/<version>/
 _url = 'https://api.kraken.com/0'
 _headers: dict[str, str] = {
    'User-Agent': 'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
 }
-# TODO: this is the only backend providing this right?
+# Broker specific ohlc schema which includes a vwap field
-# in which case we should drop it from the defaults and
+_ohlc_dtype = [
-# instead make a custom fields descr in this module!
+    ('index', int),
    ('time', int),
    ('open', float),
    ('high', float),
    ('low', float),
    ('close', float),
    ('volume', float),
    ('count', int),
    ('bar_wap', float),
 ]
 # UI components allow this to be declared such that additional
 # (historical) fields can be exposed.
 ohlc_dtype = np.dtype(_ohlc_dtype)
 _show_wap_in_history = True
 _symbol_info_translation: dict[str, str] = {
    'tick_decimals': 'pair_decimals',
@ -76,18 +77,12 @@ _symbol_info_translation: dict[str, str] = {
 def get_config() -> dict[str, Any]:
    '''
    Load our section from `piker/brokers.toml`.
-    '''
+    conf, path = config.load()
-    conf, path = config.load(
+    section = conf.get('kraken')
-        conf_name='brokers',
+
-        touch_if_dne=True,
+    if section is None:
-    )
+        log.warning(f'No config section found for kraken in {path}')
    if (section := conf.get('kraken')) is None:
        log.warning(
            f'No config section found for kraken in {path}'
        )
        return {}
    return section
@ -120,49 +115,36 @@ class InvalidKey(ValueError):
 class Client:
-    # assets and mkt pairs are key-ed by kraken's ReST response
+    # global symbol normalization table
-    # symbol-bs_mktids (we call them "X-keys" like fricking
+    _ntable: dict[str, str] = {}
-    # "XXMRZEUR"). these keys used directly since ledger endpoints
+    _atable: bidict[str, str] = bidict()
    # return transaction sets keyed with the same set!
    _Assets: dict[str, Asset] = {}
    _AssetPairs: dict[str, Pair] = {}
    # offer lookup tables for all .altname and .wsname
    # to the equivalent .xname so that various symbol-schemas
    # can be mapped to `Pair`s in the tables above.
    _altnames: dict[str, str] = {}
    _wsnames: dict[str, str] = {}
    # key-ed by `Pair.bs_fqme: str`, and thus used for search
    # allowing for lookup using piker's own FQME symbology sys.
    _pairs: dict[str, Pair] = {}
    _assets: dict[str, Asset] = {}
    def __init__(
        self,
        config: dict[str, str],
        httpx_client: httpx.AsyncClient,
        name: str = '',
        api_key: str = '',
        secret: str = ''
    ) -> None:
-
+        self._sesh = asks.Session(connections=4)
-        self._sesh: httpx.AsyncClient = httpx_client
+        self._sesh.base_location = _url
-
+        self._sesh.headers.update({
            'User-Agent':
                'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
        })
        self.conf: dict[str, str] = config
        self._pairs: list[str] = []
        self._name = name
        self._api_key = api_key
        self._secret = secret
        self.conf: dict[str, str] = config
    @property
-    def pairs(self) -> dict[str, Pair]:
+    def pairs(self) -> dict[str, Any]:
        if self._pairs is None:
            raise RuntimeError(
-                "Client didn't run `.get_mkt_pairs()` on startup?!"
+                "Make sure to run `cache_symbols()` on startup!"
            )
            # retreive and cache all symbols
        return self._pairs
@ -171,9 +153,10 @@ class Client:
        method: str,
        data: dict,
    ) -> dict[str, Any]:
-        resp: httpx.Response = await self._sesh.post(
+        resp = await self._sesh.post(
-            url=f'/public/{method}',
+            path=f'/public/{method}',
            json=data,
            timeout=float('inf')
        )
        return resproc(resp, log)
@ -184,18 +167,18 @@ class Client:
        uri_path: str
    ) -> dict[str, Any]:
        headers = {
-            'Content-Type': 'application/x-www-form-urlencoded',
+            'Content-Type':
-            'API-Key': self._api_key,
+                'application/x-www-form-urlencoded',
-            'API-Sign': get_kraken_signature(
+            'API-Key':
-                uri_path,
+                self._api_key,
-                data,
+            'API-Sign':
-                self._secret,
+                get_kraken_signature(uri_path, data, self._secret)
            ),
        }
-        resp: httpx.Response = await self._sesh.post(
+        resp = await self._sesh.post(
-            url=f'/private/{method}',
+            path=f'/private/{method}',
            data=data,
            headers=headers,
            timeout=float('inf')
        )
        return resproc(resp, log)
@ -221,81 +204,24 @@ class Client:
            'Balance',
            {},
        )
-        by_bsmktid: dict[str, dict] = resp['result']
+        by_bsuid = resp['result']
        return {
            self._atable[sym].lower(): float(bal)
            for sym, bal in by_bsuid.items()
        }
-        balances: dict = {}
+    async def get_assets(self) -> dict[str, dict]:
-        for xname, bal in by_bsmktid.items():
+        resp = await self._public('Assets', {})
-            asset: Asset = self._Assets[xname]
+        return resp['result']
-            # TODO: which KEY should we use? it's used to index
+    async def cache_assets(self) -> None:
-            # the `Account.pps: dict` ..
+        assets = self.assets = await self.get_assets()
-            key: str = asset.name.lower()
+        for bsuid, info in assets.items():
-            # TODO: should we just return a `Decimal` here
+            self._atable[bsuid] = info['altname']
            # or is the rounded version ok?
            balances[key] = round(
                float(bal),
                ndigits=dec_digits(asset.tx_tick)
            )
        return balances
    async def get_assets(
        self,
        reload: bool = False,
    ) -> dict[str, Asset]:
        '''
        Load and cache all asset infos and pack into
        our native ``Asset`` struct.
        https://docs.kraken.com/rest/#tag/Market-Data/operation/getAssetInfo
        return msg:
            "asset1": {
                "aclass": "string",
                "altname": "string",
                "decimals": 0,
                "display_decimals": 0,
                "collateral_value": 0,
                "status": "string"
            }
        '''
        if (
            not self._assets
            or reload
        ):
            resp = await self._public('Assets', {})
            assets: dict[str, dict] = resp['result']
            for bs_mktid, info in assets.items():
                altname: str = info['altname']
                aclass: str = info['aclass']
                asset = Asset(
                    name=altname,
                    atype=f'crypto_{aclass}',
                    tx_tick=digits_to_dec(info['decimals']),
                    info=info,
                )
                # NOTE: yes we keep 2 sets since kraken insists on
                # keeping 3 frickin sets bc apparently they have
                # no sane data engineers whol all like different
                # keys for their fricking symbology sets..
                self._Assets[bs_mktid] = asset
                self._assets[altname.lower()] = asset
                self._assets[altname] = asset
        # we return the "most native" set merged with our preferred
        # naming (which i guess is the "altname" one) since that's
        # what the symcache loader will be storing, and we need the
        # keys that are easiest to match against in any trade
        # records.
        return self._Assets | self._assets
    async def get_trades(
        self,
-        fetch_limit: int | None = None,
+        fetch_limit: int = 10,
    ) -> dict[str, Any]:
        '''
@ -307,10 +233,7 @@ class Client:
        trades_by_id: dict[str, Any] = {}
        for i in itertools.count():
-            if (
+            if i >= fetch_limit:
                fetch_limit
                and i >= fetch_limit
            ):
                break
            # increment 'ofs' pagination offset
@ -323,8 +246,7 @@ class Client:
            by_id = resp['result']['trades']
            trades_by_id.update(by_id)
-            # can get up to 50 results per query, see:
+            # we can get up to 50 results per query
            # https://docs.kraken.com/rest/#tag/User-Data/operation/getTradeHistory
            if (
                len(by_id) < 50
            ):
@ -354,15 +276,10 @@ class Client:
        Currently only withdrawals are supported.
        '''
-        resp = await self.endpoint(
+        xfers: list[dict] = (await self.endpoint(
            'WithdrawStatus',
            {'asset': asset},
-        )
+        ))['result']
        try:
            xfers: list[dict] = resp['result']
        except KeyError:
            log.exception(f'Kraken suxxx: {resp}')
            return []
        # eg. resp schema:
        # 'result': [{'method': 'Bitcoin', 'aclass': 'currency', 'asset':
@ -372,27 +289,20 @@ class Client:
        #     'amount': '0.00300726', 'fee': '0.00001000', 'time':
        #     1658347714, 'status': 'Success'}]}
        if xfers:
            await tractor.pause()
        trans: dict[str, Transaction] = {}
        for entry in xfers:
-            # look up the normalized name and asset info
+            # look up the normalized name
-            asset_key: str = entry['asset']
+            asset = self._atable[entry['asset']].lower()
            asset: Asset = self._Assets[asset_key]
            asset_key: str = asset.name.lower()
            # XXX: this is in the asset units (likely) so it isn't
            # quite the same as a commisions cost necessarily..)
            # TODO: also round this based on `Pair` cost precision info?
            cost = float(entry['fee'])
            # fqme: str = asset_key + '.kraken'
-            tx = Transaction(
+            tran = Transaction(
-                fqme=asset_key,  # this must map to an entry in .assets!
+                fqsn=asset + '.kraken',
                tid=entry['txid'],
                dt=pendulum.from_timestamp(entry['time']),
-                bs_mktid=f'{asset_key}{src_asset}',
+                bsuid=f'{asset}{src_asset}',
                size=-1*(
                    float(entry['amount'])
                    +
@ -403,14 +313,9 @@ class Client:
                price='NaN',
                # XXX: see note above
-                cost=cost,
+                cost=0,
                # not a trade but a withdrawal or deposit on the
                # asset (chain) system.
                etype='transfer',
            )
-            trans[tx.tid] = tx
+            trans[tran.tid] = tran
        return trans
@ -459,125 +364,69 @@ class Client:
        # txid is a transaction id given by kraken
        return await self.endpoint('CancelOrder', {"txid": reqid})
-    async def asset_pairs(
+    async def symbol_info(
        self,
-        pair_patt: str | None = None,
+        pair: Optional[str] = None,
-    ) -> dict[str, Pair] | Pair:
+    ) -> dict[str, dict[str, str]]:
        '''
        Query for a tradeable asset pair (info), or all if no input
        pattern is provided.
-        https://docs.kraken.com/rest/#tag/Market-Data/operation/getTradableAssetPairs
+        if pair is not None:
            pairs = {'pair': pair}
        else:
            pairs = None  # get all pairs
-        '''
+        resp = await self._public('AssetPairs', pairs)
-        if not self._AssetPairs:
+        err = resp['error']
-            # get all pairs by default, or filter
+        if err:
-            # to whatever pattern is provided as input.
+            symbolname = pairs['pair'] if pair else None
-            req_pairs: dict[str, str] | None = None
+            raise SymbolNotFound(f'{symbolname}.kraken')
            if pair_patt is not None:
                req_pairs = {'pair': pair_patt}
-            resp = await self._public(
+        pairs = resp['result']
                'AssetPairs',
                req_pairs,
            )
            err = resp['error']
            if err:
                raise SymbolNotFound(pair_patt)
-            # NOTE: we try to key pairs by our custom defined
+        if pair is not None:
-            # `.bs_fqme` field since we want to offer search over
+            _, data = next(iter(pairs.items()))
-            # this pattern set, callers should fill out lookup
+            return data
-            # tables for kraken's bs_mktid keys to map to these
+        else:
-            # keys!
+            return pairs
            # XXX: FURTHER kraken's data eng team decided to offer
            # 3 frickin market-pair-symbol key sets depending on
            # which frickin API is being used.
            # Example for the trading pair 'LTC<EUR'
            # - the "X-key" from rest eps 'XLTCZEUR'
            # - the "websocket key" from ws msgs is 'LTC/EUR'
            # - the "altname key" also delivered in pair info is 'LTCEUR'
            for xkey, data in resp['result'].items():
-                # NOTE: always cache in pairs tables for faster lookup
+    async def cache_symbols(
                with tractor.devx.maybe_open_crash_handler(): # as bxerr:
                    pair = Pair(xname=xkey, **data)
                # register the above `Pair` structs for all
                # key-sets/monikers: a set of 4 (frickin) tables
                # acting as a combined surjection of all possible
                # (and stupid) kraken names to their `Pair` obj.
                self._AssetPairs[xkey] = pair
                self._pairs[pair.bs_fqme] = pair
                self._altnames[pair.altname] = pair
                self._wsnames[pair.wsname] = pair
        if pair_patt is not None:
            return next(iter(self._pairs.items()))[1]
        return self._AssetPairs
    async def get_mkt_pairs(
        self,
        reload: bool = False,
    ) -> dict:
-        '''
+        if not self._pairs:
-        Load all market pair info build and cache it for downstream
+            self._pairs = await self.symbol_info()
        use.
-        Multiple pair info lookup tables (like ``._altnames:
+            ntable = {}
-        dict[str, str]``) are created for looking up the
+            for restapikey, info in self._pairs.items():
-        piker-native `Pair`-struct from any input of the three
+                ntable[restapikey] = ntable[info['wsname']] = info['altname']
        (yes, it's that idiotic..) available symbol/pair-key-sets
        that kraken frickin offers depending on the API including
        the .altname, .wsname and the weird ass default set they
        return in ReST responses .xname..
-        '''
+            self._ntable.update(ntable)
        if (
            not self._pairs
            or reload
        ):
            await self.asset_pairs()
-        return self._AssetPairs
+        return self._pairs
    async def search_symbols(
        self,
        pattern: str,
-
+        limit: int = None,
    ) -> dict[str, Any]:
-        '''
+        if self._pairs is not None:
-        Search for a symbol by "alt name"..
+            data = self._pairs
        else:
            data = await self.symbol_info()
-        It is expected that the ``Client._pairs`` table
+        matches = fuzzy.extractBests(
-        gets populated before conducting the underlying fuzzy-search
+            pattern,
-        over the pair-key set.
+            data,
        '''
        if not len(self._pairs):
            await self.get_mkt_pairs()
            assert self._pairs, '`Client.get_mkt_pairs()` was never called!?'
        matches: dict[str, Pair] = match_from_pairs(
            pairs=self._pairs,
            query=pattern.upper(),
            score_cutoff=50,
        )
-
+        # repack in dict form
-        # repack in .altname-keyed output table
+        return {item[0]['altname']: item[0] for item in matches}
        return {
            pair.altname: pair
            for pair in matches.values()
        }
    async def bars(
        self,
        symbol: str = 'XBTUSD',
        # UTC 2017-07-02 12:53:20
-        since: Union[int, datetime] | None = None,
+        since: Optional[Union[int, datetime]] = None,
        count: int = 720,  # <- max allowed per query
        as_np: bool = True,
@ -631,11 +480,11 @@ class Client:
                new_bars.append(
                    (i,) + tuple(
                        ftype(bar[j]) for j, (name, ftype) in enumerate(
-                            def_iohlcv_fields[1:]
+                            _ohlc_dtype[1:]
                        )
                    )
                )
-            array = np.array(new_bars, dtype=def_iohlcv_fields) if as_np else bars
+            array = np.array(new_bars, dtype=_ohlc_dtype) if as_np else bars
            return array
        except KeyError:
            errmsg = json['error'][0]
@ -650,9 +499,9 @@ class Client:
                raise BrokerError(errmsg)
    @classmethod
-    def to_bs_fqme(
+    def normalize_symbol(
        cls,
-        pair_str: str
+        ticker: str
    ) -> str:
        '''
        Normalize symbol names to to a 3x3 pair from the global
@ -660,45 +509,28 @@ class Client:
        the 'AssetPairs' endpoint, see methods above.
        '''
-        try:
+        ticker = cls._ntable[ticker]
-            return cls._altnames[pair_str.upper()].bs_fqme
+        return ticker.lower()
        except KeyError as ke:
            raise SymbolNotFound(f'kraken has no {ke.args[0]}')
@acm
 async def get_client() -> Client:
-    conf: dict[str, Any] = get_config()
+    conf = get_config()
-    async with httpx.AsyncClient(
+    if conf:
-        base_url=_url,
+        client = Client(
-        headers=_headers,
+            conf,
            name=conf['key_descr'],
            api_key=conf['api_key'],
            secret=conf['secret']
        )
    else:
        client = Client({})
-        # TODO: is there a way to numerate this?
+    # at startup, load all symbols, and asset info in
-        # https://www.python-httpx.org/advanced/clients/#why-use-a-client
+    # batch requests.
-        # connections=4
+    async with trio.open_nursery() as nurse:
-    ) as trio_client:
+        nurse.start_soon(client.cache_assets)
-        if conf:
+        await client.cache_symbols()
            client = Client(
                conf,
                httpx_client=trio_client,
-                # TODO: don't break these up and just do internal
+    yield client
                # conf lookups instead..
                name=conf['key_descr'],
                api_key=conf['api_key'],
                secret=conf['secret']
            )
        else:
            client = Client(
                conf={},
                httpx_client=trio_client,
            )
        # at startup, load all symbols, and asset info in
        # batch requests.
        async with trio.open_nursery() as nurse:
            nurse.start_soon(client.get_assets)
            await client.get_mkt_pairs()
        yield client
--- a/piker/brokers/kraken/broker.py
+++ b/piker/brokers/kraken/broker.py
@ -18,12 +18,14 @@
 Order api and machinery
 '''
 from collections import ChainMap, defaultdict
 from contextlib import (
    asynccontextmanager as acm,
-    aclosing,
+    contextmanager as cm,
 )
 from functools import partial
 from itertools import count
 import math
 from pprint import pformat
 import time
 from typing import (
@ -33,20 +35,18 @@ from typing import (
    Union,
 )
 from async_generator import aclosing
 from bidict import bidict
 import pendulum
 import trio
 import tractor
-from piker.accounting import (
+from piker.pp import (
    Position,
-    Account,
+    PpTable,
    Transaction,
    TransactionLedger,
    open_trade_ledger,
-    open_account,
+    open_pps,
 )
 from piker.clearing import(
    OrderDialogs,
 )
 from piker.clearing._messages import (
    Order,
@ -59,29 +59,18 @@ from piker.clearing._messages import (
    BrokerdPosition,
    BrokerdStatus,
 )
-from piker.brokers import (
+from . import log
    open_cached_client,
 )
 from piker.log import (
    get_console_log,
    get_logger,
 )
 from piker.data import open_symcache
 from .api import (
    Client,
    BrokerError,
    get_client,
 )
 from .feed import (
    get_console_log,
    open_autorecon_ws,
    NoBsWs,
    stream_messages,
 )
 from .ledger import (
    norm_trade_records,
    verify_balances,
 )
 log = get_logger(name=__name__)
 MsgUnion = Union[
    BrokerdCancel,
@ -131,7 +120,7 @@ async def handle_order_requests(
    client: Client,
    ems_order_stream: tractor.MsgStream,
    token: str,
-    apiflows: OrderDialogs,
+    apiflows: dict[int, ChainMap[dict[str, dict]]],
    ids: bidict[str, int],
    reqids2txids: dict[int, str],
@ -141,8 +130,10 @@ async def handle_order_requests(
    and deliver acks or errors.
    '''
-    # XXX: UGH, let's unify this.. with ``msgspec``!!!
+    # XXX: UGH, let's unify this.. with ``msgspec``.
-    msg: dict | Order
+    msg: dict[str, Any]
    order: BrokerdOrder
    async for msg in ems_order_stream:
        log.info(f'Rx order msg:\n{pformat(msg)}')
        match msg:
@ -180,19 +171,19 @@ async def handle_order_requests(
            case {
                'account': 'kraken.spot' as account,
-                'action': 'buy'|'sell',
+                'action': action,
-            }:
+            } if action in {'buy', 'sell'}:
                # validate
                order = BrokerdOrder(**msg)
                # logic from old `Client.submit_limit()`
                if order.oid in ids:
-                    ep: str = 'editOrder'
+                    ep = 'editOrder'
-                    reqid: int = ids[order.oid]  # integer not txid
+                    reqid = ids[order.oid]  # integer not txid
                    try:
-                        txid: str = reqids2txids[reqid]
+                        txid = reqids2txids[reqid]
                    except KeyError:
                        # XXX: not sure if this block ever gets hit now?
                        log.error('TOO FAST EDIT')
                        reqids2txids[reqid] = TooFastEdit(reqid)
@ -213,7 +204,7 @@ async def handle_order_requests(
                        }
                else:
-                    ep: str = 'addOrder'
+                    ep = 'addOrder'
                    reqid = BrokerClient.new_reqid()
                    ids[order.oid] = reqid
@ -226,12 +217,8 @@ async def handle_order_requests(
                        'type': order.action,
                    }
-                    # XXX strip any .<venue> token which should
+                    psym = order.symbol.upper()
-                    # ONLY ever be '.spot' rn, until we support
+                    pair = f'{psym[:3]}/{psym[3:]}'
                    # futes.
                    bs_fqme: str = order.symbol.replace('.spot', '')
                    psym: str = bs_fqme.upper()
                    pair: str = f'{psym[:3]}/{psym[3:]}'
                # XXX: ACK the request **immediately** before sending
                # the api side request to ensure the ems maps the oid ->
@ -266,16 +253,10 @@ async def handle_order_requests(
                } | extra
                log.info(f'Submitting WS order request:\n{pformat(req)}')
                # NOTE HOWTO, debug order requests
                #
                # if 'XRP' in pair:
                #     await tractor.pause()
                await ws.send_msg(req)
                # placehold for sanity checking in relay loop
-                apiflows.add_msg(reqid, msg)
+                apiflows[reqid].maps.append(msg)
            case _:
                account = msg.get('account')
@ -381,23 +362,22 @@ async def subscribe(
 def trades2pps(
-    acnt: Account,
+    table: PpTable,
    ledger: TransactionLedger,
    acctid: str,
    new_trans: dict[str, Transaction] = {},
-    write_storage: bool = True,
+) -> tuple[
-
+    list[BrokerdPosition],
-) -> list[BrokerdPosition]:
+    list[Transaction],
 ]:
    if new_trans:
-        updated = acnt.update_from_ledger(
+        updated = table.update_from_trans(
            new_trans,
            symcache=ledger.symcache,
        )
        log.info(f'Updated pps:\n{pformat(updated)}')
-    pp_entries, closed_pp_objs = acnt.dump_active()
+    pp_entries, closed_pp_objs = table.dump_active()
-    pp_objs: dict[Union[str, int], Position] = acnt.pps
+    pp_objs: dict[Union[str, int], Position] = table.pps
    pps: dict[int, Position]
    position_msgs: list[dict] = []
@ -411,51 +391,42 @@ def trades2pps(
                # backend suffix prefixed but when
                # reading accounts from ledgers we
                # don't need it and/or it's prefixed
-                # in the section acnt.. we should
+                # in the section table.. we should
                # just strip this from the message
                # right since `.broker` is already
                # included?
                account='kraken.' + acctid,
-                symbol=p.mkt.fqme,
+                symbol=p.symbol.front_fqsn(),
-                size=p.cumsize,
+                size=p.size,
                avg_price=p.ppu,
                currency='',
            )
            position_msgs.append(msg)
    if write_storage:
        # TODO: ideally this blocks the this task
        # as little as possible. we need to either do
        # these writes in another actor, or try out `trio`'s
        # async file IO api?
        acnt.write_config()
    return position_msgs
@tractor.context
-async def open_trade_dialog(
+async def trades_dialogue(
    ctx: tractor.Context,
-    loglevel: str = 'warning',
+    loglevel: str = None,
 ) -> AsyncIterator[dict[str, Any]]:
-    get_console_log(
+    # XXX: required to propagate ``tractor`` loglevel to ``piker`` logging
-        level=loglevel,
+    get_console_log(loglevel or tractor.current_actor().loglevel)
-        name=__name__,
+
-    )
+    async with get_client() as client:
    async with (
        # TODO: maybe bind these together and deliver
        # a tuple from `.open_cached_client()`?
        open_cached_client('kraken') as client,
        open_symcache('kraken') as symcache,
    ):
        # make ems flip to paper mode when no creds setup in
        # `brokers.toml` B0
        if not client._api_key:
-            await ctx.started('paper')
+            raise RuntimeError(
-            return
+                'Missing Kraken API key in `brokers.toml`!?!?')
        # TODO: make ems flip to paper mode via
        # some returned signal if the user only wants to use
        # the data feed or we return this?
        # else:
        #     await ctx.started(({}, ['paper']))
        # NOTE: currently we expect the user to define a "source fiat"
        # (much like the web UI let's you set an "account currency")
@ -468,7 +439,10 @@ async def open_trade_dialog(
        acc_name = 'kraken.' + acctid
        # task local msg dialog tracking
-        apiflows = OrderDialogs()
+        apiflows: defaultdict[
            int,
            ChainMap[dict[str, dict]],
        ] = defaultdict(ChainMap)
        # 2way map for ems ids to kraken int reqids..
        ids: bidict[str, int] = bidict()
@ -480,8 +454,8 @@ async def open_trade_dialog(
        # - delete the *ABSOLUTE LAST* entry from account's corresponding
        #   trade ledgers file (NOTE this MUST be the last record
        #   delivered from the api ledger),
-        # - open you ``account.kraken.spot.toml`` and find that
+        # - open you ``pps.toml`` and find that same tid and delete it
-        #   same tid and delete it from the pos's clears table,
+        #   from the pp's clears table,
        # - set this flag to `True`
        #
        # You should see an update come in after the order mode
@ -492,85 +466,154 @@ async def open_trade_dialog(
        # update things correctly.
        simulate_pp_update: bool = False
        acnt: Account
        ledger: TransactionLedger
        with (
-            open_account(
+            open_pps(
                'kraken',
                acctid,
-                write_on_exit=True,
+            ) as table,
            ) as acnt,
            open_trade_ledger(
                'kraken',
                acctid,
-                symcache=symcache,
+            ) as ledger_dict,
            ) as ledger,
        ):
-            # TODO: loading ledger entries should all be done
+            # transaction-ify the ledger entries
-            # within a newly implemented `async with open_account()
+            ledger_trans = norm_trade_records(ledger_dict)
            # as acnt` where `Account.ledger: TransactionLedger`
            # can be used to explicitily update and write the
            # offline TOML files!
            # ------ - ------
            # MOL the init sequence is:
            # - get `Account` (with presumed pre-loaded ledger done
            #   beind the scenes as part of ctx enter).
            # - pull new trades from API, update the ledger with
            #   normalized to `Transaction` entries of those
            #   records, presumably (and implicitly) update the
            #   acnt state including expiries, positions,
            #   transfers..), and finally of course existing
            #   per-asset balances.
            # - validate all pos and balances ensuring there's
            #   no seemingly noticeable discrepancies?
            # LOAD and transaction-ify the EXISTING LEDGER
            ledger_trans: dict[str, Transaction] = await norm_trade_records(
                ledger,
                client,
                api_name_set='xname',
            )
            if not acnt.pps:
                acnt.update_from_ledger(
                    ledger_trans,
                    symcache=ledger.symcache,
                )
                acnt.write_config()
            # TODO: eventually probably only load
            # as far back as it seems is not deliverd in the
            # most recent 50 trades and assume that by ordering we
-            # already have those records in the ledger?
+            # already have those records in the ledger.
-            tids2trades: dict[str, dict] = await client.get_trades()
+            tids2trades = await client.get_trades()
-            ledger.update(tids2trades)
+            ledger_dict.update(tids2trades)
-            if tids2trades:
+            api_trans = norm_trade_records(tids2trades)
                ledger.write_config()
            api_trans: dict[str, Transaction] = await norm_trade_records(
                tids2trades,
                client,
                api_name_set='xname',
            )
            # retrieve kraken reported balances
            # and do diff with ledger to determine
            # what amount of trades-transactions need
            # to be reloaded.
-            balances: dict[str, float] = await client.get_balances()
+            balances = await client.get_balances()
            for dst, size in balances.items():
                # we don't care about tracking positions
                # in the user's source fiat currency.
                if dst == src_fiat:
                    continue
-            await verify_balances(
+                def has_pp(
-                acnt,
+                    dst: str,
-                src_fiat,
+                    size: float,
                balances,
                client,
                ledger,
                ledger_trans,
                api_trans,
            )
-            # XXX NOTE: only for simulate-testing a "new fill" since
+                ) -> Position | bool:
                    src2dst: dict[str, str] = {}
                    for bsuid in table.pps:
                        try:
                            dst_name_start = bsuid.rindex(src_fiat)
                        except (
                            ValueError,   # substr not found
                        ):
                            # TODO: handle nested positions..(i.e.
                            # positions where the src fiat was used to
                            # buy some other dst which was furhter used
                            # to buy another dst..)
                            log.warning(
                                f'No src fiat {src_fiat} found in {bsuid}?'
                            )
                            continue
                        _dst = bsuid[:dst_name_start]
                        if _dst != dst:
                            continue
                        src2dst[src_fiat] = dst
                    for src, dst in src2dst.items():
                        pair = f'{dst}{src_fiat}'
                        pp = table.pps.get(pair)
                        if (
                            pp
                            and math.isclose(pp.size, size)
                        ):
                            return pp
                        elif (
                            size == 0
                            and pp.size
                        ):
                            log.warning(
                                f'`kraken` account says you have  a ZERO '
                                f'balance for {bsuid}:{pair}\n'
                                f'but piker seems to think `{pp.size}`\n'
                                'This is likely a discrepancy in piker '
                                'accounting if the above number is'
                                "large,' though it's likely to due lack"
                                "f tracking xfers fees.."
                            )
                            return pp
                    return False
                pos = has_pp(dst, size)
                if not pos:
                    # we have a balance for which there is no pp
                    # entry? so we have to likely update from the
                    # ledger.
                    updated = table.update_from_trans(ledger_trans)
                    log.info(f'Updated pps from ledger:\n{pformat(updated)}')
                    pos = has_pp(dst, size)
                    if (
                        not pos
                        and not simulate_pp_update
                    ):
                        # try reloading from API
                        table.update_from_trans(api_trans)
                        pos = has_pp(dst, size)
                        if not pos:
                            # get transfers to make sense of abs balances.
                            # NOTE: we do this after ledger and API
                            # loading since we might not have an entry
                            # in the ``pps.toml`` for the necessary pair
                            # yet and thus this likely pair grabber will
                            # likely fail.
                            likely_pair = {
                                bsuid[:3]: bsuid
                                for bsuid in table.pps
                            }.get(dst)
                            if not likely_pair:
                                raise ValueError(
                                    'Could not find a position pair in '
                                    'ledger for likely widthdrawal '
                                    f'candidate: {dst}'
                                )
                            if likely_pair:
                                # this was likely pp that had a withdrawal
                                # from the dst asset out of the account.
                                xfer_trans = await client.get_xfers(
                                    dst,
                                    src_asset=likely_pair[3:],
                                )
                                if xfer_trans:
                                    updated = table.update_from_trans(
                                        xfer_trans,
                                        cost_scalar=1,
                                    )
                                    log.info(
                                        'Updated {dst} from transfers:\n'
                                        f'{pformat(updated)}'
                                    )
                        if has_pp(dst, size):
                            raise ValueError(
                                'Could not reproduce balance:\n'
                                f'dst: {dst}, {size}\n'
                            )
            # only for simulate-testing a "new fill" since
            # otherwise we have to actually conduct a live clear.
            if simulate_pp_update:
                tid = list(tids2trades)[0]
@ -578,28 +621,20 @@ async def open_trade_dialog(
                # stage a first reqid of `0`
                reqids2txids[0] = last_trade_dict['ordertxid']
-            ppmsgs: list[BrokerdPosition] = trades2pps(
+            ppmsgs = trades2pps(
-                acnt,
+                table,
                ledger,
                acctid,
            )
            # sync with EMS delivering pps and accounts
            await ctx.started((ppmsgs, [acc_name]))
            # TODO: ideally this blocks the this task
            # as little as possible. we need to either do
            # these writes in another actor, or try out `trio`'s
            # async file IO api?
            acnt.write_config()
            # Get websocket token for authenticated data stream
            # Assert that a token was actually received.
            resp = await client.endpoint('GetWebSocketsToken', {})
-            if err := resp.get('error'):
+            err = resp.get('error')
            if err:
                raise BrokerError(err)
-            # resp token for ws init
+            token = resp['result']['token']
            token: str = resp['result']['token']
            ws: NoBsWs
            async with (
@ -614,6 +649,8 @@ async def open_trade_dialog(
                aclosing(stream_messages(ws)) as stream,
                trio.open_nursery() as nurse,
            ):
                stream = stream_messages(ws)
                # task for processing inbound requests from ems
                nurse.start_soon(
                    handle_order_requests,
@ -628,35 +665,32 @@ async def open_trade_dialog(
                # enter relay loop
                await handle_order_updates(
-                    client=client,
+                    ws,
-                    ws=ws,
+                    stream,
-                    ws_stream=stream,
+                    ems_stream,
-                    ems_stream=ems_stream,
+                    apiflows,
-                    apiflows=apiflows,
+                    ids,
-                    ids=ids,
+                    reqids2txids,
-                    reqids2txids=reqids2txids,
+                    table,
-                    acnt=acnt,
+                    api_trans,
-                    ledger=ledger,
+                    acctid,
-                    acctid=acctid,
+                    acc_name,
-                    acc_name=acc_name,
+                    token,
                    token=token,
                )
 async def handle_order_updates(
    client: Client,  # only for pairs table needed in ledger proc
    ws: NoBsWs,
    ws_stream: AsyncIterator,
    ems_stream: tractor.MsgStream,
-    apiflows: OrderDialogs,
+    apiflows: dict[int, ChainMap[dict[str, dict]]],
    ids: bidict[str, int],
    reqids2txids: bidict[int, str],
-    acnt: Account,
+    table: PpTable,
    # transaction records which will be updated
    # on new trade clearing events (aka order "fills")
-    ledger: TransactionLedger,
+    ledger_trans: dict[str, Transaction],
    # ledger_trans: dict[str, Transaction],
    acctid: str,
    acc_name: str,
    token: str,
@ -665,8 +699,8 @@ async def handle_order_updates(
    '''
    Main msg handling loop for all things order management.
-    This code is broken out to make the context explicit and state
+    This code is broken out to make the context explicit and state variables
-    variables defined in the signature clear to the reader.
+    defined in the signature clear to the reader.
    '''
    async for msg in ws_stream:
@ -674,7 +708,7 @@ async def handle_order_updates(
            # TODO: turns out you get the fill events from the
            # `openOrders` before you get this, so it might be better
-            # to do all fill/status/pos updates in that sub and just use
+            # to do all fill/status/pp updates in that sub and just use
            # this one for ledger syncs?
            # For eg. we could take the "last 50 trades" and do a diff
@ -716,8 +750,7 @@ async def handle_order_updates(
                    # if tid not in ledger_trans
                }
                for tid, trade in trades.items():
-                    # assert tid not in ledger_trans
+                    assert tid not in ledger_trans
                    assert tid not in ledger
                    txid = trade['ordertxid']
                    reqid = trade.get('userref')
@ -760,25 +793,17 @@ async def handle_order_updates(
                    )
                    await ems_stream.send(status_msg)
-                new_trans = await norm_trade_records(
+                new_trans = norm_trade_records(trades)
-                    trades,
+                ppmsgs = trades2pps(
-                    client,
+                    table,
-                    api_name_set='wsname',
+                    acctid,
                    new_trans,
                )
                ppmsgs: list[BrokerdPosition] = trades2pps(
                    acnt=acnt,
                    ledger=ledger,
                    acctid=acctid,
                    new_trans=new_trans,
                )
                # ppmsgs = trades2pps(
                #     acnt,
                #     acctid,
                #     new_trans,
                # )
                for pp_msg in ppmsgs:
                    await ems_stream.send(pp_msg)
                ledger_trans.update(new_trans)
            # process and relay order state change events
            # https://docs.kraken.com/websockets/#message-openOrders
            case [
@ -820,9 +845,8 @@ async def handle_order_updates(
                    # 'vol_exec': exec_vlm}  # 0.0000
                    match update_msg:
-                        # EMS-unknown pre-exising-submitted LIVE
+                        # EMS-unknown LIVE order that needs to be
-                        # order that needs to be delivered and
+                        # delivered and loaded on the client-side.
                        # loaded on the client-side.
                        case {
                            'userref': reqid,
                            'descr': {
@ -841,7 +865,7 @@ async def handle_order_updates(
                            ids.inverse.get(reqid) is None
                        ):
                            # parse out existing live order
-                            fqme = pair.replace('/', '').lower() + '.spot'
+                            fqsn = pair.replace('/', '').lower()
                            price = float(price)
                            size = float(vol)
@ -868,14 +892,14 @@ async def handle_order_updates(
                                    action=action,
                                    exec_mode='live',
                                    oid=oid,
-                                    symbol=fqme,
+                                    symbol=fqsn,
                                    account=acc_name,
                                    price=price,
                                    size=size,
                                ),
                                src='kraken',
                            )
-                            apiflows.add_msg(reqid, status_msg.to_dict())
+                            apiflows[reqid].maps.append(status_msg.to_dict())
                            await ems_stream.send(status_msg)
                            continue
@ -1011,7 +1035,7 @@ async def handle_order_updates(
                                ),
                            )
-                            apiflows.add_msg(reqid, update_msg)
+                            apiflows[reqid].maps.append(update_msg)
                            await ems_stream.send(resp)
                        # fill msg.
@ -1090,8 +1114,9 @@ async def handle_order_updates(
                    )
                    continue
-                # update the msg history
+                # update the msg chain
-                apiflows.add_msg(reqid, event)
+                chain = apiflows[reqid]
                chain.maps.append(event)
                if status == 'error':
                    # any of ``{'add', 'edit', 'cancel'}``
@ -1101,18 +1126,11 @@ async def handle_order_updates(
                        f'Failed to {action} order {reqid}:\n'
                        f'{errmsg}'
                    )
                    # if tractor._state.debug_mode():
                    #     await tractor.pause()
                    symbol: str = 'N/A'
                    if chain := apiflows.get(reqid):
                        symbol: str = chain.get('symbol', 'N/A')
                    await ems_stream.send(BrokerdError(
                        oid=oid,
                        # XXX: use old reqid in case it changed?
                        reqid=reqid,
-                        symbol=symbol,
+                        symbol=chain.get('symbol', 'N/A'),
                        reason=f'Failed {action}:\n{errmsg}',
                        broker_details=event
@ -1137,3 +1155,56 @@ async def handle_order_updates(
                        })
            case _:
                log.warning(f'Unhandled trades update msg: {msg}')
 def norm_trade_records(
    ledger: dict[str, Any],
 ) -> dict[str, Transaction]:
    records: dict[str, Transaction] = {}
    for tid, record in ledger.items():
        size = float(record.get('vol')) * {
            'buy': 1,
            'sell': -1,
        }[record['type']]
        # we normalize to kraken's `altname` always..
        bsuid = norm_sym = Client.normalize_symbol(record['pair'])
        records[tid] = Transaction(
            fqsn=f'{norm_sym}.kraken',
            tid=tid,
            size=size,
            price=float(record['price']),
            cost=float(record['fee']),
            dt=pendulum.from_timestamp(float(record['time'])),
            bsuid=bsuid,
            # XXX: there are no derivs on kraken right?
            # expiry=expiry,
        )
    return records
@cm
 def open_ledger(
    acctid: str,
    trade_entries: list[dict[str, Any]],
 ) -> set[Transaction]:
    '''
    Write recent session's trades to the user's (local) ledger file.
    '''
    with open_trade_ledger(
        'kraken',
        acctid,
    ) as ledger:
        yield ledger
        # update on exit
        ledger.update(trade_entries)
--- a/piker/brokers/kraken/feed.py
+++ b/piker/brokers/kraken/feed.py
@ -18,44 +18,81 @@
 Real-time and historical data feed endpoints.
 '''
-from contextlib import (
+from contextlib import asynccontextmanager as acm
    asynccontextmanager as acm,
    aclosing,
 )
 from datetime import datetime
 from typing import (
-    AsyncGenerator,
+    Any,
    Callable,
    Optional,
    Callable,
 )
 import time
 from async_generator import aclosing
 from fuzzywuzzy import process as fuzzy
 import numpy as np
 import pendulum
 from trio_typing import TaskStatus
 import tractor
 import trio
-from piker.accounting._mktinfo import (
+from piker._cacheables import open_cached_client
    MktPair,
 )
 from piker.brokers import (
    open_cached_client,
 )
 from piker.brokers._util import (
    BrokerError,
    DataThrottle,
    DataUnavailable,
 )
-from piker.types import Struct
+from piker.log import get_console_log
-from piker.data.validate import FeedInit
+from piker.data import ShmArray
 from piker.data.types import Struct
 from piker.data._web_bs import open_autorecon_ws, NoBsWs
 from . import log
 from .api import (
-    log,
+    Client,
 )
 from .symbols import get_mkt_info
-class OHLC(Struct, frozen=True):
+# https://www.kraken.com/features/api#get-tradable-pairs
 class Pair(Struct):
    altname: str  # alternate pair name
    wsname: str  # WebSocket pair name (if available)
    aclass_base: str  # asset class of base component
    base: str  # asset id of base component
    aclass_quote: str  # asset class of quote component
    quote: str  # asset id of quote component
    lot: str  # volume lot size
    cost_decimals: int
    costmin: float
    pair_decimals: int  # scaling decimal places for pair
    lot_decimals: int  # scaling decimal places for volume
    # amount to multiply lot volume by to get currency volume
    lot_multiplier: float
    # array of leverage amounts available when buying
    leverage_buy: list[int]
    # array of leverage amounts available when selling
    leverage_sell: list[int]
    # fee schedule array in [volume, percent fee] tuples
    fees: list[tuple[int, float]]
    # maker fee schedule array in [volume, percent fee] tuples (if on
    # maker/taker)
    fees_maker: list[tuple[int, float]]
    fee_volume_currency: str  # volume discount currency
    margin_call: str  # margin call level
    margin_stop: str  # stop-out/liquidation margin level
    ordermin: float  # minimum order volume for pair
    tick_size: float # min price step size
    status: str
    short_position_limit: float
    long_position_limit: float
 class OHLC(Struct):
    '''
    Description of the flattened OHLC quote format.
@ -66,8 +103,6 @@ class OHLC(Struct, frozen=True):
    chan_id: int  # internal kraken id
    chan_name: str  # eg. ohlc-1  (name-interval)
    pair: str  # fx pair
    # unpacked from array
    time: float  # Begin time of interval, in seconds since epoch
    etime: float  # End time of interval, in seconds since epoch
    open: float  # Open price of interval
@ -77,6 +112,8 @@ class OHLC(Struct, frozen=True):
    vwap: float  # Volume weighted average price within interval
    volume: float  # Accumulated volume **within interval**
    count: int  # Number of trades within interval
    # (sampled) generated tick data
    ticks: list[Any] = []
 async def stream_messages(
@ -89,9 +126,26 @@ async def stream_messages(
    though a single async generator.
    '''
-    last_hb: float = 0
+    too_slow_count = last_hb = 0
    while True:
        with trio.move_on_after(5) as cs:
            msg = await ws.recv_msg()
        # trigger reconnection if heartbeat is laggy
        if cs.cancelled_caught:
            too_slow_count += 1
            if too_slow_count > 20:
                log.warning(
                    "Heartbeat is too slow, resetting ws connection")
                await ws._connect()
                too_slow_count = 0
                continue
    async for msg in ws:
        match msg:
            case {'event': 'heartbeat'}:
                now = time.time()
@ -116,99 +170,90 @@ async def process_data_feed_msgs(
    Parse and pack data feed messages.
    '''
-    async with aclosing(stream_messages(ws)) as ws_stream:
+    async for msg in stream_messages(ws):
-        async for msg in ws_stream:
+        match msg:
-            match msg:
+            case {
-                case {
+                'errorMessage': errmsg
-                    'errorMessage': errmsg
+            }:
-                }:
+                raise BrokerError(errmsg)
                    raise BrokerError(errmsg)
-                case {
+            case {
-                    'event': 'subscriptionStatus',
+                'event': 'subscriptionStatus',
-                } as sub:
+            } as sub:
-                    log.info(
+                log.info(
-                        'WS subscription is active:\n'
+                    'WS subscription is active:\n'
-                        f'{sub}'
+                    f'{sub}'
                )
                continue
            case [
                chan_id,
                *payload_array,
                chan_name,
                pair
            ]:
                if 'ohlc' in chan_name:
                    ohlc = OHLC(
                        chan_id,
                        chan_name,
                        pair,
                        *payload_array[0]
                    )
-                    continue
+                    ohlc.typecast()
                    yield 'ohlc', ohlc
-                case [
+                elif 'spread' in chan_name:
                    chan_id,
                    *payload_array,
                    chan_name,
                    pair
                ]:
                    if 'ohlc' in chan_name:
                        array: list = payload_array[0]
                        ohlc = OHLC(
                            chan_id,
                            chan_name,
                            pair,
                            *map(float, array[:-1]),
                            count=array[-1],
                        )
                        yield 'ohlc', ohlc.copy()
-                    elif 'spread' in chan_name:
+                    bid, ask, ts, bsize, asize = map(
                        float, payload_array[0])
-                        bid, ask, ts, bsize, asize = map(
+                    # TODO: really makes you think IB has a horrible API...
-                            float, payload_array[0])
+                    quote = {
                        'symbol': pair.replace('/', ''),
                        'ticks': [
                            {'type': 'bid', 'price': bid, 'size': bsize},
                            {'type': 'bsize', 'price': bid, 'size': bsize},
-                        # TODO: really makes you think IB has a horrible API...
+                            {'type': 'ask', 'price': ask, 'size': asize},
-                        quote = {
+                            {'type': 'asize', 'price': ask, 'size': asize},
-                            'symbol': pair.replace('/', ''),
+                        ],
-                            'ticks': [
+                    }
-                                {'type': 'bid', 'price': bid, 'size': bsize},
+                    yield 'l1', quote
                                {'type': 'bsize', 'price': bid, 'size': bsize},
-                                {'type': 'ask', 'price': ask, 'size': asize},
+                # elif 'book' in msg[-2]:
-                                {'type': 'asize', 'price': ask, 'size': asize},
+                #     chan_id, *payload_array, chan_name, pair = msg
-                            ],
+                #     print(msg)
                        }
                        yield 'l1', quote
-                    # elif 'book' in msg[-2]:
+            case _:
-                    #     chan_id, *payload_array, chan_name, pair = msg
+                print(f'UNHANDLED MSG: {msg}')
-                    #     print(msg)
+                # yield msg
                case {
                    'connectionID': conid,
                    'event': 'systemStatus',
                    'status': 'online',
                    'version': ver,
                }:
                    log.info(
                        f'Established {ver} ws connection with id: {conid}'
                    )
                    continue
                case _:
                    print(f'UNHANDLED MSG: {msg}')
                    # yield msg
-def normalize(ohlc: OHLC) -> dict:
+def normalize(
-    '''
+    ohlc: OHLC,
    Norm an `OHLC` msg to piker's minimal (live-)quote schema.
-    '''
+) -> dict:
    quote = ohlc.to_dict()
    quote['broker_ts'] = quote['time']
    quote['brokerd_ts'] = time.time()
    quote['symbol'] = quote['pair'] = quote['pair'].replace('/', '')
    quote['last'] = quote['close']
    quote['bar_wap'] = ohlc.vwap
-    return quote
+
    # seriously eh? what's with this non-symmetry everywhere
    # in subscription systems...
    # XXX: piker style is always lowercases symbols.
    topic = quote['pair'].replace('/', '').lower()
    # print(quote)
    return topic, quote
@acm
 async def open_history_client(
-    mkt: MktPair,
+    symbol: str,
-) -> AsyncGenerator[Callable, None]:
+) -> tuple[Callable, int]:
    symbol: str = mkt.bs_mktid
    # TODO implement history getter for the new storage layer.
    async with open_cached_client('kraken') as client:
@ -278,20 +323,45 @@ async def stream_quotes(
    ``pairs`` must be formatted <crypto_symbol>/<fiat_symbol>.
    '''
    # XXX: required to propagate ``tractor`` loglevel to piker logging
    get_console_log(loglevel or tractor.current_actor().loglevel)
-    ws_pairs: list[str] = []
+    ws_pairs = {}
-    init_msgs: list[FeedInit] = []
+    sym_infos = {}
-    async with (
+    async with open_cached_client('kraken') as client, send_chan as send_chan:
        send_chan as send_chan,
    ):
        for sym_str in symbols:
            mkt, pair = await get_mkt_info(sym_str)
            init_msgs.append(
                FeedInit(mkt_info=mkt)
            )
-            ws_pairs.append(pair.wsname)
+        # keep client cached for real-time section
        for sym in symbols:
            # transform to upper since piker style is always lower
            sym = sym.upper()
            sym_info = await client.symbol_info(sym)
            try:
                si = Pair(**sym_info)  # validation
            except TypeError:
                fields_diff = set(sym_info) - set(Pair.__struct_fields__)
                raise TypeError(
                    f'Missing msg fields {fields_diff}'
                )
            syminfo = si.to_dict()
            syminfo['price_tick_size'] = 1 / 10**si.pair_decimals
            syminfo['lot_tick_size'] = 1 / 10**si.lot_decimals
            syminfo['asset_type'] = 'crypto'
            sym_infos[sym] = syminfo
            ws_pairs[sym] = si.wsname
        symbol = symbols[0].lower()
        init_msgs = {
            # pass back token, and bool, signalling if we're the writer
            # and that history has been written
            symbol: {
                'symbol_info': sym_infos[sym],
                'shm_write_opts': {'sum_tick_vml': False},
                'fqsn': sym,
            },
        }
        @acm
        async def subscribe(ws: NoBsWs):
@ -302,7 +372,7 @@ async def stream_quotes(
            # https://github.com/krakenfx/kraken-wsclient-py/blob/master/kraken_wsclient_py/kraken_wsclient_py.py#L188
            ohlc_sub = {
                'event': 'subscribe',
-                'pair': ws_pairs,
+                'pair': list(ws_pairs.values()),
                'subscription': {
                    'name': 'ohlc',
                    'interval': 1,
@ -318,7 +388,7 @@ async def stream_quotes(
            # trade data (aka L1)
            l1_sub = {
                'event': 'subscribe',
-                'pair': ws_pairs,
+                'pair': list(ws_pairs.values()),
                'subscription': {
                    'name': 'spread',
                    # 'depth': 10}
@ -333,7 +403,7 @@ async def stream_quotes(
            # unsub from all pairs on teardown
            if ws.connected():
                await ws.send_msg({
-                    'pair': ws_pairs,
+                    'pair': list(ws_pairs.values()),
                    'event': 'unsubscribe',
                    'subscription': ['ohlc', 'spread'],
                })
@ -348,68 +418,83 @@ async def stream_quotes(
            open_autorecon_ws(
                'wss://ws.kraken.com/',
                fixture=subscribe,
                reset_after=20,
            ) as ws,
            # avoid stream-gen closure from breaking trio..
            # NOTE: not sure this actually works XD particularly
            # if we call `ws._connect()` manally in the streaming
            # async gen..
            aclosing(process_data_feed_msgs(ws)) as msg_gen,
        ):
            # pull a first quote and deliver
            typ, ohlc_last = await anext(msg_gen)
-            quote = normalize(ohlc_last)
+            topic, quote = normalize(ohlc_last)
            task_status.started((init_msgs,  quote))
            # lol, only "closes" when they're margin squeezing clients ;P
            feed_is_live.set()
            # keep start of last interval for volume tracking
-            last_interval_start: float = ohlc_last.etime
+            last_interval_start = ohlc_last.etime
            # start streaming
-            topic: str = mkt.bs_fqme
+            async for typ, ohlc in msg_gen:
-            async for typ, quote in msg_gen:
+
-                match typ:
+                if typ == 'ohlc':
                    # TODO: can get rid of all this by using
-                    # ``trades`` subscription..? Not sure why this
+                    # ``trades`` subscription...
                    # wasn't used originally? (music queues) zoltannn..
                    # https://docs.kraken.com/websockets/#message-trade
                    case 'ohlc':
                        # generate tick values to match time & sales pane:
                        # https://trade.kraken.com/charts/KRAKEN:BTC-USD?period=1m
                        volume = quote.volume
-                        # new OHLC sample interval
+                    # generate tick values to match time & sales pane:
-                        if quote.etime > last_interval_start:
+                    # https://trade.kraken.com/charts/KRAKEN:BTC-USD?period=1m
-                            last_interval_start: float = quote.etime
+                    volume = ohlc.volume
                            tick_volume: float = volume
-                        else:
+                    # new OHLC sample interval
-                            # this is the tick volume *within the interval*
+                    if ohlc.etime > last_interval_start:
-                            tick_volume: float = volume - ohlc_last.volume
+                        last_interval_start = ohlc.etime
                        tick_volume = volume
-                        ohlc_last = quote
+                    else:
-                        last = quote.close
+                        # this is the tick volume *within the interval*
                        tick_volume = volume - ohlc_last.volume
-                        quote = normalize(quote)
+                    ohlc_last = ohlc
-                        ticks = quote.setdefault(
+                    last = ohlc.close
                            'ticks',
                            [],
                        )
                        if tick_volume:
                            ticks.append({
                                'type': 'trade',
                                'price': last,
                                'size': tick_volume,
                            })
-                    case 'l1':
+                    if tick_volume:
-                        # passthrough quote msg
+                        ohlc.ticks.append({
-                        pass
+                            'type': 'trade',
                            'price': last,
                            'size': tick_volume,
                        })
-                    case _:
+                    topic, quote = normalize(ohlc)
-                        log.warning(f'Unknown WSS message: {typ}, {quote}')
+
                elif typ == 'l1':
                    quote = ohlc
                    topic = quote['symbol'].lower()
                await send_chan.send({topic: quote})
@tractor.context
 async def open_symbol_search(
    ctx: tractor.Context,
 ) -> Client:
    async with open_cached_client('kraken') as client:
        # load all symbols locally for fast search
        cache = await client.cache_symbols()
        await ctx.started(cache)
        async with ctx.open_stream() as stream:
            async for pattern in stream:
                matches = fuzzy.extractBests(
                    pattern,
                    cache,
                    score_cutoff=50,
                )
                # repack in dict form
                await stream.send(
                    {item[0]['altname']: item[0]
                     for item in matches}
                )
--- a/piker/brokers/kraken/ledger.py
+++ b/piker/brokers/kraken/ledger.py
@ -1,269 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Trade transaction accounting and normalization.
 '''
 import math
 from pprint import pformat
 from typing import (
    Any,
 )
 import pendulum
 from piker.accounting import (
    Transaction,
    Position,
    Account,
    get_likely_pair,
    TransactionLedger,
    # MktPair,
 )
 from piker.types import Struct
 from piker.data import (
    SymbologyCache,
 )
 from .api import (
    log,
    Client,
    Pair,
 )
 # from .feed import get_mkt_info
 def norm_trade(
    tid: str,
    record: dict[str, Any],
    # this is the dict that was returned from
    # `Client.get_mkt_pairs()` and when running offline ledger
    # processing from `.accounting`, this will be the table loaded
    # into `SymbologyCache.pairs`.
    pairs: dict[str, Struct],
    symcache: SymbologyCache | None = None,
 ) -> Transaction:
    size: float = float(record.get('vol')) * {
        'buy': 1,
        'sell': -1,
    }[record['type']]
    # NOTE: this value may be either the websocket OR the rest schema
    # so we need to detect the key format and then choose the
    # correct symbol lookup table to evetually get a ``Pair``..
    # See internals of `Client.asset_pairs()` for deats!
    src_pair_key: str = record['pair']
    # XXX: kraken's data engineering is soo bad they require THREE
    # different pair schemas (more or less seemingly tied to
    # transport-APIs)..LITERALLY they return different market id
    # pairs in the ledger endpoints vs. the websocket event subs..
    # lookup pair using appropriately provided tabled depending
    # on API-key-schema..
    pair: Pair = pairs[src_pair_key]
    fqme: str = pair.bs_fqme.lower() + '.kraken'
    return Transaction(
        fqme=fqme,
        tid=tid,
        size=size,
        price=float(record['price']),
        cost=float(record['fee']),
        dt=pendulum.from_timestamp(float(record['time'])),
        bs_mktid=pair.bs_mktid,
    )
 async def norm_trade_records(
    ledger: dict[str, Any],
    client: Client,
    api_name_set: str = 'xname',
 ) -> dict[str, Transaction]:
    '''
    Loop through an input ``dict`` of trade records
    and convert them to ``Transactions``.
    '''
    records: dict[str, Transaction] = {}
    for tid, record in ledger.items():
        # manual_fqme: str = f'{bs_mktid.lower()}.kraken'
        # mkt: MktPair = (await get_mkt_info(manual_fqme))[0]
        # fqme: str = mkt.fqme
        # assert fqme == manual_fqme
        pairs: dict[str, Pair] = {
            'xname': client._AssetPairs,
            'wsname': client._wsnames,
            'altname': client._altnames,
        }[api_name_set]
        records[tid] = norm_trade(
            tid,
            record,
            pairs=pairs,
        )
    return records
 def has_pp(
    acnt: Account,
    src_fiat: str,
    dst: str,
    size: float,
 ) -> Position | None:
    src2dst: dict[str, str] = {}
    for bs_mktid in acnt.pps:
        likely_pair = get_likely_pair(
            src_fiat,
            dst,
            bs_mktid,
        )
        if likely_pair:
            src2dst[src_fiat] = dst
    for src, dst in src2dst.items():
        pair: str = f'{dst}{src_fiat}'
        pos: Position = acnt.pps.get(pair)
        if (
            pos
            and math.isclose(pos.size, size)
        ):
            return pos
        elif (
            size == 0
            and pos.size
        ):
            log.warning(
                f'`kraken` account says you have  a ZERO '
                f'balance for {bs_mktid}:{pair}\n'
                f'but piker seems to think `{pos.size}`\n'
                'This is likely a discrepancy in piker '
                'accounting if the above number is'
                "large,' though it's likely to due lack"
                "f tracking xfers fees.."
            )
            return pos
    return None  # indicate no entry found
 # TODO: factor most of this "account updating from txns" into the
 # the `Account` impl so has to provide for hiding the mostly
 # cross-provider updates from txn sets
 async def verify_balances(
    acnt: Account,
    src_fiat: str,
    balances: dict[str, float],
    client: Client,
    ledger: TransactionLedger,
    ledger_trans: dict[str, Transaction],  # from toml
    api_trans: dict[str, Transaction],  # from API
    simulate_pp_update: bool = False,
 ) -> None:
    for dst, size in balances.items():
        # we don't care about tracking positions
        # in the user's source fiat currency.
        if (
            dst == src_fiat
            or not any(
                dst in bs_mktid for bs_mktid in acnt.pps
            )
        ):
            log.warning(
                f'Skipping balance `{dst}`:{size} for position calcs!'
            )
            continue
        # we have a balance for which there is no pos entry
        # - we have to likely update from the ledger?
        if not has_pp(acnt, src_fiat, dst, size):
            updated = acnt.update_from_ledger(
                ledger_trans,
                symcache=ledger.symcache,
            )
            log.info(f'Updated pps from ledger:\n{pformat(updated)}')
            # FIRST try reloading from API records
            if (
                not has_pp(acnt, src_fiat, dst, size)
                and not simulate_pp_update
            ):
                acnt.update_from_ledger(
                    api_trans,
                    symcache=ledger.symcache,
                )
                # get transfers to make sense of abs
                # balances.
                # NOTE: we do this after ledger and API
                # loading since we might not have an
                # entry in the
                # ``account.kraken.spot.toml`` for the
                # necessary pair yet and thus this
                # likely pair grabber will likely fail.
                if not has_pp(acnt, src_fiat, dst, size):
                    for bs_mktid in acnt.pps:
                        likely_pair: str | None = get_likely_pair(
                            src_fiat,
                            dst,
                            bs_mktid,
                        )
                        if likely_pair:
                            break
                    else:
                        raise ValueError(
                            'Could not find a position pair in '
                            'ledger for likely widthdrawal '
                            f'candidate: {dst}'
                        )
                    # this was likely pos that had a withdrawal
                    # from the dst asset out of the account.
                    if likely_pair:
                        xfer_trans = await client.get_xfers(
                            dst,
                            # TODO: not all src assets are
                            # 3 chars long...
                            src_asset=likely_pair[3:],
                        )
                        if xfer_trans:
                            updated = acnt.update_from_ledger(
                                xfer_trans,
                                cost_scalar=1,
                                symcache=ledger.symcache,
                            )
                            log.info(
                                f'Updated {dst} from transfers:\n'
                                f'{pformat(updated)}'
                            )
                if has_pp(acnt, src_fiat, dst, size):
                    raise ValueError(
                        'Could not reproduce balance:\n'
                        f'dst: {dst}, {size}\n'
                    )
--- a/piker/brokers/kraken/symbols.py
+++ b/piker/brokers/kraken/symbols.py
@ -1,210 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Symbology defs and search.
 '''
 from decimal import Decimal
 import tractor
 from piker._cacheables import (
    async_lifo_cache,
 )
 from piker.accounting._mktinfo import (
    digits_to_dec,
 )
 from piker.brokers import (
    open_cached_client,
    SymbolNotFound,
 )
 from piker.types import Struct
 from piker.accounting._mktinfo import (
    Asset,
    MktPair,
    unpack_fqme,
 )
 class Pair(Struct):
    '''
    A tradable asset pair as schema-defined by,
    https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs
    '''
    xname: str  # idiotic bs_mktid equiv i guess?
    altname: str  # alternate pair name
    wsname: str  # WebSocket pair name (if available)
    aclass_base: str  # asset class of base component
    base: str  # asset id of base component
    aclass_quote: str  # asset class of quote component
    quote: str  # asset id of quote component
    lot: str  # volume lot size
    cost_decimals: int
    pair_decimals: int  # scaling decimal places for pair
    lot_decimals: int  # scaling decimal places for volume
    # amount to multiply lot volume by to get currency volume
    lot_multiplier: float
    # array of leverage amounts available when buying
    leverage_buy: list[int]
    # array of leverage amounts available when selling
    leverage_sell: list[int]
    # fee schedule array in [volume, percent fee] tuples
    fees: list[tuple[int, float]]
    # maker fee schedule array in [volume, percent fee] tuples (if on
    # maker/taker)
    fees_maker: list[tuple[int, float]]
    fee_volume_currency: str  # volume discount currency
    margin_call: str  # margin call level
    margin_stop: str  # stop-out/liquidation margin level
    ordermin: float  # minimum order volume for pair
    tick_size: float  # min price step size
    status: str
    costmin: str|None = None  # XXX, only some mktpairs?
    short_position_limit: float = 0
    long_position_limit: float = float('inf')
    # TODO: should we make this a literal NamespacePath ref?
    ns_path: str = 'piker.brokers.kraken:Pair'
    @property
    def bs_mktid(self) -> str:
        '''
        Kraken seems to index it's market symbol sets in
        transaction ledgers using the key returned from rest
        queries.. so use that since apparently they can't
        make up their minds on a better key set XD
        '''
        return self.xname
    @property
    def price_tick(self) -> Decimal:
        return digits_to_dec(self.pair_decimals)
    @property
    def size_tick(self) -> Decimal:
        return digits_to_dec(self.lot_decimals)
    @property
    def bs_dst_asset(self) -> str:
        dst, _ = self.wsname.split('/')
        return dst
    @property
    def bs_src_asset(self) -> str:
        _, src = self.wsname.split('/')
        return src
    @property
    def bs_fqme(self) -> str:
        '''
        Basically the `.altname` but with special '.' handling and
        `.SPOT` suffix appending (for future multi-venue support).
        '''
        dst, src = self.wsname.split('/')
        # XXX: omg for stupid shite like ETH2.S/ETH..
        dst = dst.replace('.', '-')
        return f'{dst}{src}.SPOT'
@tractor.context
 async def open_symbol_search(ctx: tractor.Context) -> None:
    async with open_cached_client('kraken') as client:
        # load all symbols locally for fast search
        cache = await client.get_mkt_pairs()
        await ctx.started(cache)
        async with ctx.open_stream() as stream:
            async for pattern in stream:
                await stream.send(
                    await client.search_symbols(pattern)
                )
@async_lifo_cache()
 async def get_mkt_info(
    fqme: str,
 ) -> tuple[MktPair, Pair]:
    '''
    Query for and return a `MktPair` and backend-native `Pair` (or
    wtv else) info.
    If more then one fqme is provided return a ``dict`` of native
    key-strs to `MktPair`s.
    '''
    venue: str = 'spot'
    expiry: str = ''
    if '.kraken' not in fqme:
        fqme += '.kraken'
    broker, pair, venue, expiry = unpack_fqme(fqme)
    venue: str = venue or 'spot'
    if venue.lower() != 'spot':
        raise SymbolNotFound(
            'kraken only supports spot markets right now!\n'
            f'{fqme}\n'
        )
    async with open_cached_client('kraken') as client:
        # uppercase since kraken bs_mktid is always upper
        # bs_fqme, _, broker = fqme.partition('.')
        # pair_str: str = bs_fqme.upper()
        pair_str: str = f'{pair}.{venue}'
        pair: Pair | None = client._pairs.get(pair_str.upper())
        if not pair:
            bs_fqme: str = client.to_bs_fqme(pair_str)
            pair: Pair = client._pairs[bs_fqme]
        if not (assets := client._assets):
            assets: dict[str, Asset] = await client.get_assets()
        dst_asset: Asset = assets[pair.bs_dst_asset]
        src_asset: Asset = assets[pair.bs_src_asset]
        mkt = MktPair(
            dst=dst_asset,
            src=src_asset,
            price_tick=pair.price_tick,
            size_tick=pair.size_tick,
            bs_mktid=pair.bs_mktid,
            expiry=expiry,
            venue=venue or 'spot',
            # TODO: futes
            # _atype=_atype,
            broker='kraken',
        )
        return mkt, pair
--- a/piker/brokers/kucoin.py
+++ b/piker/brokers/kucoin.py
--- a/piker/brokers/questrade.py
+++ b/piker/brokers/questrade.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -37,32 +37,16 @@ import tractor
 from async_generator import asynccontextmanager
 import numpy as np
 import wrapt
 # TODO, port to `httpx`/`trio-websocket` whenver i get back to
 # writing a proper ws-api streamer for this backend (since the data
 # feeds are free now) as per GH feat-req:
 # https://github.com/pikers/piker/issues/509
 #
 import asks
 from ..calc import humanize, percent_change
-from . import open_cached_client
+from .._cacheables import open_cached_client, async_lifo_cache
 from piker._cacheables import async_lifo_cache
 from .. import config
 from ._util import resproc, BrokerError, SymbolNotFound
-from piker.log import (
+from ..log import get_logger, colorize_json, get_console_log
    colorize_json,
    get_console_log,
 )
 from piker.log import (
    get_logger,
 )
-log = get_logger(
+log = get_logger(__name__)
    name=__name__,
 )
 _use_practice_account = False
 _refresh_token_ep = 'https://{}login.questrade.com/oauth2/'
@ -1211,10 +1195,7 @@ async def stream_quotes(
    # feed_type: str = 'stock',
 ) -> AsyncGenerator[str, Dict[str, Any]]:
    # XXX: required to propagate ``tractor`` loglevel to piker logging
-    get_console_log(
+    get_console_log(loglevel)
        level=loglevel,
        name=__name__,
    )
    async with open_cached_client('questrade') as client:
        if feed_type == 'stock':
--- a/piker/brokers/robinhood.py
+++ b/piker/brokers/robinhood.py
@ -27,19 +27,11 @@ from typing import List
 from async_generator import asynccontextmanager
 import asks
-from ._util import (
+from ..log import get_logger
-    resproc,
+from ._util import resproc, BrokerError
-    BrokerError,
+from ..calc import percent_change
 )
 from piker.calc import percent_change
 from piker.log import (
    get_logger,
 )
 log = get_logger(
    name=__name__,
 )
 log = get_logger(__name__)
 _service_ep = 'https://api.robinhood.com'
@ -73,10 +65,8 @@ class Client:
        self.api = _API(self._sess)
    def _zip_in_order(self, symbols: [str], quotes: List[dict]):
-        return {
+        return {quote.get('symbol', sym) if quote else sym: quote
-            quote.get('symbol', sym) if quote else sym: quote
+                for sym, quote in zip(symbols, results_dict)}
            for sym, quote in zip(symbols, quotes)
        }
    async def quote(self, symbols: [str]):
        """Retrieve quotes for a list of ``symbols``.
--- a/piker/clearing/README.rst
+++ b/piker/clearing/README.rst
@ -1,49 +0,0 @@
 piker.clearing
 ______________
 trade execution-n-control subsys for both live and paper trading as
 well as algo-trading manual override/interaction across any backend
 broker and data provider.
 avail UIs
 *********
 order ctl
 ---------
 the `piker.clearing` subsys is exposed mainly though
 the `piker chart` GUI as a "chart trader" style UX and
 is automatically enabled whenever a chart is opened.
 .. ^TODO, more prose here!
 the "manual" order control features are exposed via the
 `piker.ui.order_mode` API and can pretty much always be
 used (at least) in simulated-trading mode, aka "paper"-mode, and
 the micro-manual is as follows:
 ``order_mode`` (
    edge triggered activation by any of the following keys,
    ``mouse-click`` on y-level to submit at that price
    ):
    - ``f``/ ``ctl-f`` to stage buy
    - ``d``/ ``ctl-d`` to stage sell
    - ``a`` to stage alert
 ``search_mode`` (
    ``ctl-l`` or ``ctl-space`` to open,
    ``ctl-c`` or ``ctl-space`` to close
    ) :
    - begin typing to have symbol search automatically lookup
      symbols from all loaded backend (broker) providers
    - arrow keys and mouse click to navigate selection
    - vi-like ``ctl-[hjkl]`` for navigation
 position (pp) mgmt
 ------------------
 you can also configure your position allocation limits from the
 sidepane.
 .. ^TODO, explain and provide tut once more refined!
--- a/piker/clearing/init.py
+++ b/piker/clearing/init.py
@ -18,38 +18,9 @@
 Market machinery for order executions, book, management.
 """
-from ..log import get_logger
+from ._client import open_ems
 from ._client import (
    open_ems,
    OrderClient,
 )
 from ._ems import (
    open_brokerd_dialog,
 )
 from ._util import OrderDialogs
 from ._messages import(
    Order,
    Status,
    Cancel,
    # TODO: deprecate these and replace end-2-end with
    # client-side-dialog set above B)
    # https://github.com/pikers/piker/issues/514
    BrokerdPosition
 )
 __all__ = [
    'FeeModel',
    'open_ems',
    'OrderClient',
    'open_brokerd_dialog',
    'OrderDialogs',
    'Order',
    'Status',
    'Cancel',
    'BrokerdPosition'
 ]
 log = get_logger(__name__)
--- a/piker/accounting/_allocate.py
+++ b/piker/accounting/_allocate.py
@ -23,9 +23,9 @@ from typing import Optional
 from bidict import bidict
-from ._pos import Position
+from ..data._source import Symbol
-from . import MktPair
+from ..data.types import Struct
-from piker.types import Struct
+from ..pp import Position
 _size_units = bidict({
@ -42,7 +42,7 @@ SizeUnit = Enum(
 class Allocator(Struct):
-    mkt: MktPair
+    symbol: Symbol
    # TODO: if we ever want ot support non-uniform entry-slot-proportion
    # "sizes"
@ -114,24 +114,24 @@ class Allocator(Struct):
        depending on position / order entry config.
        '''
-        mkt: MktPair = self.mkt
+        sym = self.symbol
-        ld: int = mkt.size_tick_digits
+        ld = sym.lot_size_digits
        size_unit = self.size_unit
-        live_size = live_pp.cumsize
+        live_size = live_pp.size
        abs_live_size = abs(live_size)
-        abs_startup_size = abs(startup_pp.cumsize)
+        abs_startup_size = abs(startup_pp.size)
        u_per_slot, currency_per_slot = self.step_sizes()
        if size_unit == 'units':
-            slot_size: float = u_per_slot
+            slot_size = u_per_slot
-            l_sub_pp: float = self.units_limit - abs_live_size
+            l_sub_pp = self.units_limit - abs_live_size
        elif size_unit == 'currency':
-            live_cost_basis: float = abs_live_size * live_pp.ppu
+            live_cost_basis = abs_live_size * live_pp.ppu
-            slot_size: float = currency_per_slot / price
+            slot_size = currency_per_slot / price
-            l_sub_pp: float = (self.currency_limit - live_cost_basis) / price
+            l_sub_pp = (self.currency_limit - live_cost_basis) / price
        else:
            raise ValueError(
@ -141,14 +141,8 @@ class Allocator(Struct):
        # an entry (adding-to or starting a pp)
        if (
            live_size == 0
-            or (
+            or (action == 'buy' and live_size > 0)
-                action == 'buy'
+            or action == 'sell' and live_size < 0
                and live_size > 0
            )
            or (
                action == 'sell'
                and live_size < 0
            )
        ):
            order_size = min(
                slot_size,
@ -184,7 +178,7 @@ class Allocator(Struct):
            order_size = max(slotted_pp, slot_size)
            if (
-                abs_live_size < slot_size
+                abs_live_size < slot_size or
                # NOTE: front/back "loading" heurstic:
                # if the remaining pp is in between 0-1.5x a slot's
@ -193,17 +187,14 @@ class Allocator(Struct):
                # **without** going past a net-zero pp. if the pp is
                # > 1.5x a slot size, then front load: exit a slot's and
                # expect net-zero to be acquired on the final exit.
-                or slot_size < pp_size < round((1.5*slot_size), ndigits=ld)
+                slot_size < pp_size < round((1.5*slot_size), ndigits=ld) or
                or (
-                    # underlying requires discrete (int) units (eg. stocks)
+                # underlying requires discrete (int) units (eg. stocks)
-                    # and thus our slot size (based on our limit) would
+                # and thus our slot size (based on our limit) would
-                    # exit a fractional unit's worth so, presuming we aren't
+                # exit a fractional unit's worth so, presuming we aren't
-                    # supporting a fractional-units-style broker, we need
+                # supporting a fractional-units-style broker, we need
-                    # exit the final unit.
+                # exit the final unit.
-                    ld == 0
+                ld == 0 and abs_live_size == 1
                    and abs_live_size == 1
                )
            ):
                order_size = abs_live_size
@ -212,12 +203,13 @@ class Allocator(Struct):
            # compute a fractional slots size to display
            slots_used = self.slots_used(
                Position(
-                    mkt=mkt,
+                    symbol=sym,
-                    bs_mktid=mkt.bs_mktid,
+                    size=order_size,
                    ppu=price,
                    bsuid=sym,
                )
            )
        # TODO: render an actual ``Executable`` type here?
        return {
            'size': abs(round(order_size, ndigits=ld)),
            'size_digits': ld,
@ -239,7 +231,7 @@ class Allocator(Struct):
        Calc and return the number of slots used by this ``Position``.
        '''
-        abs_pp_size = abs(pp.cumsize)
+        abs_pp_size = abs(pp.size)
        if self.size_unit == 'currency':
            # live_currency_size = size or (abs_pp_size * pp.ppu)
@ -257,7 +249,7 @@ class Allocator(Struct):
 def mk_allocator(
-    mkt: MktPair,
+    symbol: Symbol,
    startup_pp: Position,
    # default allocation settings
@ -284,6 +276,6 @@ def mk_allocator(
    defaults.update(user_def)
    return Allocator(
-        mkt=mkt,
+        symbol=symbol,
        **defaults,
    )
--- a/piker/clearing/_client.py
+++ b/piker/clearing/_client.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -25,108 +25,67 @@ from typing import TYPE_CHECKING
 import trio
 import tractor
-from tractor.trionics import (
+from tractor.trionics import broadcast_receiver
    broadcast_receiver,
    collapse_eg,
 )
-from ._util import (
+from ..log import get_logger
-    log,  # sub-sys logger
+from ..data.types import Struct
-)
+from .._daemon import maybe_open_emsd
-from piker.types import Struct
+from ._messages import Order, Cancel
-from ..service import maybe_open_emsd
+from ..brokers import get_brokermod
 from ._messages import (
    Order,
    Cancel,
    BrokerdPosition,
 )
 if TYPE_CHECKING:
    from ._messages import (
        BrokerdPosition,
        Status,
    )
-class OrderClient(Struct):
+log = get_logger(__name__)
    '''
    EMS-client-side order book ctl and tracking.
-    (A)sync API for submitting orders and alerts to the `emsd` service;
+
-    this is the main control for execution management from client code.
+class OrderBook(Struct):
    '''EMS-client-side order book ctl and tracking.
    A style similar to "model-view" is used here where this api is
    provided as a supervised control for an EMS actor which does all the
    hard/fast work of talking to brokers/exchanges to conduct
    executions.
    Currently, this is mostly for keeping local state to match the EMS
    and use received events to trigger graphics updates.
    '''
    # IPC stream to `emsd` actor
    _ems_stream: tractor.MsgStream
    # mem channels used to relay order requests to the EMS daemon
-    _to_relay_task: trio.abc.SendChannel
+    _to_ems: trio.abc.SendChannel
-    _from_sync_order_client: trio.abc.ReceiveChannel
+    _from_order_book: trio.abc.ReceiveChannel
    # history table
    _sent_orders: dict[str, Order] = {}
-    def send_nowait(
+    def send(
        self,
        msg: Order | dict,
-    ) -> dict | Order:
+    ) -> dict:
        '''
        Sync version of ``.send()``.
        '''
        self._sent_orders[msg.oid] = msg
-        self._to_relay_task.send_nowait(msg)
+        self._to_ems.send_nowait(msg)
        return msg
-    async def send(
+    def send_update(
        self,
        msg: Order | dict,
    ) -> dict | Order:
        '''
        Send a new order msg async to the `emsd` service.
        '''
        self._sent_orders[msg.oid] = msg
        await self._ems_stream.send(msg)
        return msg
    def update_nowait(
        self,
        uuid: str,
        **data: dict,
    ) -> dict:
        '''
        Sync version of ``.update()``.
        '''
        cmd = self._sent_orders[uuid]
        msg = cmd.copy(update=data)
        self._sent_orders[uuid] = msg
-        self._to_relay_task.send_nowait(msg)
+        self._to_ems.send_nowait(msg)
-        return msg
+        return cmd
-    async def update(
+    def cancel(self, uuid: str) -> bool:
-        self,
+        """Cancel an order (or alert) in the EMS.
        uuid: str,
        **data: dict,
    ) -> dict:
        '''
        Update an existing order dialog with a msg updated from
        ``update`` kwargs.
-        '''
+        """
        cmd = self._sent_orders[uuid]
        msg = cmd.copy(update=data)
        self._sent_orders[uuid] = msg
        await self._ems_stream.send(msg)
        return msg
    def _mk_cancel_msg(
        self,
        uuid: str,
    ) -> Cancel:
        cmd = self._sent_orders.get(uuid)
        if not cmd:
            log.error(
@ -134,76 +93,77 @@ class OrderClient(Struct):
                f'Maybe there is a stale entry or line?\n'
                f'You should report this as a bug!'
            )
-            return
+        msg = Cancel(
        fqme = str(cmd.symbol)
        return Cancel(
            oid=uuid,
-            symbol=fqme,
+            symbol=cmd.symbol,
        )
        self._to_ems.send_nowait(msg)
 _orders: OrderBook = None
 def get_orders(
    emsd_uid: tuple[str, str] = None
 ) -> OrderBook:
    """"
    OrderBook singleton factory per actor.
    """
    if emsd_uid is not None:
        # TODO: read in target emsd's active book on startup
        pass
    global _orders
    if _orders is None:
        size = 100
        tx, rx = trio.open_memory_channel(size)
        brx = broadcast_receiver(rx, size)
        # setup local ui event streaming channels for request/resp
        # streamging with EMS daemon
        _orders = OrderBook(
            _to_ems=tx,
            _from_order_book=brx,
        )
-    def cancel_nowait(
+    return _orders
        self,
        uuid: str,
    ) -> None:
        '''
        Sync version of ``.cancel()``.
        '''
        self._to_relay_task.send_nowait(
            self._mk_cancel_msg(uuid)
        )
    async def cancel(
        self,
        uuid: str,
    ) -> bool:
        '''
        Cancel an already existintg order (or alert) dialog.
        '''
        await self._ems_stream.send(
            self._mk_cancel_msg(uuid)
        )
 # TODO: we can get rid of this relay loop once we move
 # order_mode inputs to async code!
 async def relay_order_cmds_from_sync_code(
 async def relay_orders_from_sync_code(
    client: OrderClient,
    symbol_key: str,
    to_ems_stream: tractor.MsgStream,
 ) -> None:
-    '''
+    """
-    Order submission relay task: deliver orders sent from synchronous (UI)
+    Order streaming task: deliver orders transmitted from UI
-    code to the EMS via ``OrderClient._from_sync_order_client``.
+    to downstream consumers.
    This is run in the UI actor (usually the one running Qt but could be
    any other client service code). This process simply delivers order
-    messages to the above ``_to_relay_task`` send channel (from sync code using
+    messages to the above ``_to_ems`` send channel (from sync code using
    ``.send_nowait()``), these values are pulled from the channel here
    and relayed to any consumer(s) that called this function using
    a ``tractor`` portal.
    This effectively makes order messages look like they're being
    "pushed" from the parent to the EMS where local sync code is likely
-    doing the pushing from some non-async UI handler.
+    doing the pushing from some UI.
-    '''
+    """
-    async with (
+    book = get_orders()
-        client._from_sync_order_client.subscribe() as sync_order_cmds
+    async with book._from_order_book.subscribe() as orders_stream:
-    ):
+        async for cmd in orders_stream:
        async for cmd in sync_order_cmds:
            sym = cmd.symbol
-            msg = pformat(cmd.to_dict())
+            msg = pformat(cmd)
            if sym == symbol_key:
                log.info(f'Send order cmd:\n{msg}')
                # send msg over IPC / wire
                await to_ems_stream.send(cmd)
            else:
                log.warning(
                    f'Ignoring unmatched order cmd for {sym} != {symbol_key}:'
@ -213,53 +173,77 @@ async def relay_orders_from_sync_code(
@acm
 async def open_ems(
-    fqme: str,
+    fqsn: str,
    mode: str = 'live',
    loglevel: str = 'warning',
 ) -> tuple[
-    OrderClient,  # client
+    OrderBook,
-    tractor.MsgStream,  # order ctl stream
+    tractor.MsgStream,
    dict[
        # brokername, acctid
        tuple[str, str],
-        dict[str, BrokerdPosition],
+        list[BrokerdPosition],
    ],
    list[str],
    dict[str, Status],
 ]:
    '''
-    (Maybe) spawn an EMS-daemon (emsd), deliver an `OrderClient` for
+    Spawn an EMS daemon and begin sending orders and receiving
-    requesting orders/alerts and a `trades_stream` which delivers all
+    alerts.
    response-msgs.
-    This is a "client side" entrypoint which may spawn the `emsd` service
+    This EMS tries to reduce most broker's terrible order entry apis to
-    if it can't be discovered and generally speaking is the lowest level
+    a very simple protocol built on a few easy to grok and/or
-    broker control client-API.
+    "rantsy" premises:
    - most users will prefer "dark mode" where orders are not submitted
      to a broker until and execution condition is triggered
      (aka client-side "hidden orders")
    - Brokers over-complicate their apis and generally speaking hire
      poor designers to create them. We're better off using creating a super
      minimal, schema-simple, request-event-stream protocol to unify all the
      existing piles of shit (and shocker, it'll probably just end up
      looking like a decent crypto exchange's api)
    - all order types can be implemented with client-side limit orders
    - we aren't reinventing a wheel in this case since none of these
      brokers are exposing FIX protocol; it is they doing the re-invention.
    TODO: make some fancy diagrams using mermaid.io
    the possible set of responses from the stream  is currently:
    - 'dark_submitted', 'broker_submitted'
    - 'dark_cancelled', 'broker_cancelled'
    - 'dark_executed', 'broker_executed'
    - 'broker_filled'
    '''
-    # TODO: prolly hand in the `MktPair` instance directly here as well!
+    # wait for service to connect back to us signalling
-    from piker.accounting import unpack_fqme
+    # ready for order commands
-    broker, mktep, venue, suffix = unpack_fqme(fqme)
+    book = get_orders()
-    async with maybe_open_emsd(
+    from ..data._source import unpack_fqsn
-        broker,
+    broker, symbol, suffix = unpack_fqsn(fqsn)
-        # XXX NOTE, LOL so this determines the daemon `emsd` loglevel
+
-        # then FYI.. that's kinda wrong no?
+    async with maybe_open_emsd(broker) as portal:
-        # -[ ] shouldn't it be set by `pikerd -l` or no?
+
-        # -[ ] would make a lot more sense to have a subsys ctl for
+        mod = get_brokermod(broker)
-        #     levels.. like `-l emsd.info` or something?
+        if (
-        loglevel=loglevel,
+            not getattr(mod, 'trades_dialogue', None)
-    ) as portal:
+            or mode == 'paper'
        ):
            mode = 'paper'
        from ._ems import _emsd_main
        async with (
            # connect to emsd
            portal.open_context(
                _emsd_main,
-                fqme=fqme,
+                fqsn=fqsn,
                exec_mode=mode,
                loglevel=loglevel,
            ) as (
                ctx,
@ -273,39 +257,18 @@ async def open_ems(
            # open 2-way trade command stream
            ctx.open_stream() as trades_stream,
        ):
            size: int = 100  # what should this be?
            tx, rx = trio.open_memory_channel(size)
            brx = broadcast_receiver(rx, size)
            # setup local ui event streaming channels for request/resp
            # streamging with EMS daemon
            client = OrderClient(
                _ems_stream=trades_stream,
                _to_relay_task=tx,
                _from_sync_order_client=brx,
            )
            client._ems_stream = trades_stream
            # start sync code order msg delivery task
-            async with (
+            async with trio.open_nursery() as n:
-                collapse_eg(),
+                n.start_soon(
-                trio.open_nursery() as tn,
+                    relay_order_cmds_from_sync_code,
-            ):
+                    fqsn,
                tn.start_soon(
                    relay_orders_from_sync_code,
                    client,
                    fqme,
                    trades_stream
                )
                yield (
-                    client,
+                    book,
                    trades_stream,
                    positions,
                    accounts,
                    dialogs,
                )
                # stop the sync-msg-relay task on exit.
                tn.cancel_scope.cancel()
--- a/piker/clearing/_ems.py
+++ b/piker/clearing/_ems.py
--- a/piker/clearing/_messages.py
+++ b/piker/clearing/_messages.py
@ -18,15 +18,39 @@
 Clearing sub-system message and protocols.
 """
-from __future__ import annotations
+# from collections import (
-from decimal import Decimal
+#     ChainMap,
 #     deque,
 # )
 from typing import (
    Optional,
    Literal,
 )
-from msgspec import field
+from ..data._source import Symbol
 from ..data.types import Struct
-from piker.types import Struct
+
 # TODO: a composite for tracking msg flow on 2-legged
 # dialogs.
 # class Dialog(ChainMap):
 #     '''
 #     Msg collection abstraction to easily track the state changes of
 #     a msg flow in one high level, query-able and immutable construct.
 #     The main use case is to query data from a (long-running)
 #     msg-transaction-sequence
 #     '''
 #     def update(
 #         self,
 #         msg,
 #     ) -> None:
 #         self.maps.insert(0, msg.to_dict())
 #     def flatten(self) -> dict:
 #         return dict(self)
 # TODO: ``msgspec`` stuff worth paying attention to:
@ -68,22 +92,13 @@ class Order(Struct):
    # internal ``emdsd`` unique "order id"
    oid: str  # uuid4
-    # TODO: figure out how to optionally typecast this to `MktPair`?
+    symbol: str | Symbol
    symbol: str  # | MktPair
    account: str  # should we set a default as '' ?
-    # https://docs.python.org/3/library/decimal.html#decimal-objects
+    price: float
    #
    # ?TODO? decimal usage throughout?
    # -[ ] possibly leverage the `Encoder(decimal_format='number')`
    #  bit?
    # |_https://jcristharif.com/msgspec/supported-types.html#decimal
    # -[ ] should we also use it for .size?
    #
    price: Decimal
    size: float  # -ve is "sell", +ve is "buy"
-    brokers: list[str] = []
+    brokers: Optional[list[str]] = []
 class Cancel(Struct):
@ -123,7 +138,7 @@ class Status(Struct):
    # this maps normally to the ``BrokerdOrder.reqid`` below, an id
    # normally allocated internally by the backend broker routing system
-    reqid: int | str | None = None
+    reqid: Optional[int | str] = None
    # the (last) source order/request msg if provided
    # (eg. the Order/Cancel which causes this msg) and
@ -136,7 +151,7 @@ class Status(Struct):
    # event that wasn't originated by piker's emsd (eg. some external
    # trading system which does it's own order control but that you
    # might want to "track" using piker UIs/systems).
-    src: str | None = None
+    src: Optional[str] = None
    # set when a cancel request msg was set for this order flow dialog
    # but the brokerd dialog isn't yet in a cancelled state.
@ -147,18 +162,6 @@ class Status(Struct):
    brokerd_msg: dict = {}
 class Error(Status):
    resp: str = 'error'
    # TODO: allow re-wrapping from existing (last) status?
    @classmethod
    def from_status(
        cls,
        msg: Status,
    ) -> Error:
        ...
 # ---------------
 # emsd -> brokerd
 # ---------------
@ -176,7 +179,7 @@ class BrokerdCancel(Struct):
    # for setting a unique order id then this value will be relayed back
    # on the emsd order request stream as the ``BrokerdOrderAck.reqid``
    # field
-    reqid: int | str | None = None
+    reqid: Optional[int | str] = None
    action: str = 'cancel'
@ -186,8 +189,8 @@ class BrokerdOrder(Struct):
    account: str
    time_ns: int
-    symbol: str  # fqme
+    symbol: str  # fqsn
-    price: Decimal
+    price: float
    size: float
    # TODO: if we instead rely on a +ve/-ve size to determine
@ -200,7 +203,7 @@ class BrokerdOrder(Struct):
    # for setting a unique order id then this value will be relayed back
    # on the emsd order request stream as the ``BrokerdOrderAck.reqid``
    # field
-    reqid: int | str | None = None
+    reqid: Optional[int | str] = None
 # ---------------
@ -222,27 +225,24 @@ class BrokerdOrderAck(Struct):
    # emsd id originally sent in matching request msg
    oid: str
    # TODO: do we need this?
    account: str = ''
    name: str = 'ack'
 class BrokerdStatus(Struct):
    time_ns: int
    reqid: int | str
    time_ns: int
    status: Literal[
        'open',
        'canceled',
        'fill',
        'pending',
-        # 'error',  # NOTE: use `BrokerdError`
+        'error',
        'closed',
    ]
    name: str = 'status'
-    oid: str = ''
+    account: str
-    # TODO: do we need this?
+    name: str = 'status'
    account: str | None = None,
    filled: float = 0.0
    reason: str = ''
    remaining: float = 0.0
@ -250,31 +250,31 @@ class BrokerdStatus(Struct):
    # external: bool = False
    # XXX: not required schema as of yet
-    broker_details: dict = field(default_factory=lambda: {
+    broker_details: dict = {
        'name': '',
-    })
+    }
 class BrokerdFill(Struct):
    '''
-    A single message indicating a "fill-details" event from the
+    A single message indicating a "fill-details" event from the broker
-    broker if avaiable.
+    if avaiable.
    '''
    # brokerd timestamp required for order mode arrow placement on x-axis
    # TODO: maybe int if we force ns?
    # we need to normalize this somehow since backends will use their
    # own format and likely across many disparate epoch clocks...
    time_ns: int
    broker_time: float
    reqid: int | str
    time_ns: int
    # order exeuction related
    size: float
    price: float
    name: str = 'fill'
-    action: str | None = None
+    action: Optional[str] = None
    broker_details: dict = {}  # meta-data (eg. commisions etc.)
@ -285,30 +285,23 @@ class BrokerdError(Struct):
    This is still a TODO thing since we're not sure how to employ it yet.
    '''
    oid: str
    symbol: str
    reason: str
    # TODO: drop this right?
    symbol: str | None = None
    oid: str | None = None
    # if no brokerd order request was actually submitted (eg. we errored
    # at the ``pikerd`` layer) then there will be ``reqid`` allocated.
-    reqid: str | None = None
+    reqid: Optional[int | str] = None
    name: str = 'error'
    broker_details: dict = {}
 # TODO: yeah, so we REALLY need to completely deprecate
 # this and use the `.accounting.Position` msg-type instead..
 # -[ ] an alternative might be to add a `Position.summary() ->
 #   `PositionSummary`-msg that we generate since `Position` has a lot
 #   of fields by default we likely don't want to send over the wire?
 class BrokerdPosition(Struct):
-    '''
+    '''Position update event from brokerd.
    Position update event from brokerd.
    '''
    broker: str
    account: str
    symbol: str
@ -316,4 +309,3 @@ class BrokerdPosition(Struct):
    avg_price: float
    currency: str = ''
    name: str = 'position'
    bs_mktid: str|int|None = None
--- a/piker/clearing/_paper_engine.py
+++ b/piker/clearing/_paper_engine.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -14,24 +14,21 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-'''
+"""
-Fake trading: a full forward testing simulation engine.
+Fake trading for forward testing.
-We can real-time emulate any mkt conditions you want bruddr B)
+"""
 Just slide us the model que quieres..
 '''
 from collections import defaultdict
-from contextlib import asynccontextmanager as acm
+from contextlib import asynccontextmanager
 from datetime import datetime
 from operator import itemgetter
 import itertools
 from pprint import pformat
 import time
 from typing import (
    Any,
    Optional,
    Callable,
 )
 from types import ModuleType
 import uuid
 from bidict import bidict
@ -39,30 +36,16 @@ import pendulum
 import trio
 import tractor
-from piker.brokers import get_brokermod
+from .. import data
-from piker.service import find_service
+from ..data._source import Symbol
-from piker.accounting import (
+from ..data.types import Struct
-    Account,
+from ..pp import (
    MktPair,
    Position,
    Transaction,
    TransactionLedger,
    open_account,
    open_trade_ledger,
    unpack_fqme,
 )
 from piker.data import (
    Feed,
    SymbologyCache,
    iterticks,
    open_feed,
    open_symcache,
 )
 from piker.types import Struct
 from piker.log import (
    get_console_log,
    get_logger,
 )
 from ..data._normalize import iterticks
 from ..data._source import unpack_fqsn
 from ..log import get_logger
 from ._messages import (
    BrokerdCancel,
    BrokerdOrder,
@ -73,7 +56,8 @@ from ._messages import (
    BrokerdError,
 )
-log = get_logger(name=__name__)
+
 log = get_logger(__name__)
 class PaperBoi(Struct):
@ -85,17 +69,16 @@ class PaperBoi(Struct):
    '''
    broker: str
    ems_trades_stream: tractor.MsgStream
    acnt: Account
    ledger: TransactionLedger
    fees: Callable
    # map of paper "live" orders which be used
    # to simulate fills based on paper engine settings
    _buys: defaultdict[str, bidict]
    _sells: defaultdict[str, bidict]
    _reqids: bidict
-    _mkts: dict[str, MktPair] = {}
+    _positions: dict[str, Position]
    _trade_ledger: dict[str, Any]
    # init edge case L1 spread
    last_ask: tuple[float, float] = (float('inf'), 0)  # price, size
@ -108,7 +91,7 @@ class PaperBoi(Struct):
        price: float,
        action: str,
        size: float,
-        reqid: str | None,
+        reqid: Optional[str],
    ) -> int:
        '''
@ -132,12 +115,9 @@ class PaperBoi(Struct):
        # for dark orders since we want the dark_executed
        # to trigger first thus creating a lookup entry
        # in the broker trades event processing loop
-        await trio.sleep(0.01)
+        await trio.sleep(0.05)
-        if (
+        if action == 'sell':
            action == 'sell'
            and size > 0
        ):
            size = -size
        msg = BrokerdStatus(
@ -199,7 +179,7 @@ class PaperBoi(Struct):
            self._sells[symbol].pop(oid, None)
        # TODO: net latency model
-        await trio.sleep(0.01)
+        await trio.sleep(0.05)
        msg = BrokerdStatus(
            status='canceled',
@ -213,7 +193,7 @@ class PaperBoi(Struct):
    async def fake_fill(
        self,
-        fqme: str,
+        fqsn: str,
        price: float,
        size: float,
        action: str,  # one of {'buy', 'sell'}
@ -232,7 +212,7 @@ class PaperBoi(Struct):
        '''
        # TODO: net latency model
-        await trio.sleep(0.01)
+        await trio.sleep(0.05)
        fill_time_ns = time.time_ns()
        fill_time_s = time.time()
@ -254,6 +234,8 @@ class PaperBoi(Struct):
        log.info(f'Fake filling order:\n{fill_msg}')
        await self.ems_trades_stream.send(fill_msg)
        self._trade_ledger.update(fill_msg.to_dict())
        if order_complete:
            msg = BrokerdStatus(
                reqid=reqid,
@ -266,59 +248,42 @@ class PaperBoi(Struct):
            )
            await self.ems_trades_stream.send(msg)
-        # NOTE: for paper we set the "bs_mktid" as just the fqme since
+        # lookup any existing position
-        # we don't actually have any unique backend symbol ourselves
+        key = fqsn.rstrip(f'.{self.broker}')
-        # other then this thing, our fqme address.
+        pp = self._positions.setdefault(
-        bs_mktid: str = fqme
+            fqsn,
-        if fees := self.fees:
+            Position(
-            cost: float = fees(price, size)
+                Symbol(
-        else:
+                    key=key,
-            cost: float = 0
+                    broker_info={self.broker: {}},
-
+                ),
                size=size,
                ppu=price,
                bsuid=key,
            )
        )
        t = Transaction(
-            fqme=fqme,
+            fqsn=fqsn,
            tid=oid,
            size=size,
            price=price,
-            cost=cost,
+            cost=0,  # TODO: cost model
            dt=pendulum.from_timestamp(fill_time_s),
-            bs_mktid=bs_mktid,
+            bsuid=key,
        )
-
+        pp.add_clear(t)
        # update in-mem ledger and pos table
        self.ledger.update_from_t(t)
        self.acnt.update_from_ledger(
            {oid: t},
            symcache=self.ledger._symcache,
            # XXX when a backend has no symcache support yet we can
            # simply pass in the gmi() retreived table created
            # during init :o
            _mktmap_table=self._mkts,
        )
        # transmit pp msg to ems
        pp: Position = self.acnt.pps[bs_mktid]
        # TODO, this will break if `require_only=True` was passed to
        # `.update_from_ledger()`
        pp_msg = BrokerdPosition(
            broker=self.broker,
            account='paper',
-            symbol=fqme,
+            symbol=fqsn,
            size=pp.cumsize,
            avg_price=pp.ppu,
            # TODO: we need to look up the asset currency from
            # broker info. i guess for crypto this can be
            # inferred from the pair?
-            # currency=bs_mktid,
+            currency='',
            size=pp.size,
            avg_price=pp.ppu,
        )
        # write all updates to filesys immediately
        # (adds latency but that works for simulation anyway)
        self.ledger.write_config()
        self.acnt.write_config()
        await self.ems_trades_stream.send(pp_msg)
@ -347,7 +312,6 @@ async def simulate_fills(
    # this stream may eventually contain multiple symbols
    async for quotes in quote_stream:
        for sym, quote in quotes.items():
            # print(sym)
            for tick in iterticks(
                quote,
                # dark order price filter(s)
@ -456,7 +420,7 @@ async def simulate_fills(
                        # clearing price would have filled entirely
                        await client.fake_fill(
-                            fqme=sym,
+                            fqsn=sym,
                            # todo slippage to determine fill price
                            price=tick_price,
                            size=size,
@ -504,7 +468,6 @@ async def handle_order_requests(
                    BrokerdOrderAck(
                        oid=order.oid,
                        reqid=reqid,
                        account='paper'
                    )
                )
@ -512,7 +475,7 @@ async def handle_order_requests(
                reqid = await client.submit_limit(
                    oid=order.oid,
                    symbol=f'{order.symbol}.{client.broker}',
-                    price=float(order.price),
+                    price=order.price,
                    action=order.action,
                    size=order.size,
                    # XXX: by default 0 tells ``ib_insync`` methods that
@ -548,210 +511,80 @@ _sells: defaultdict[
        tuple[float, float, str, str],  # order info
    ]
 ] = defaultdict(bidict)
 _positions: dict[str, Position] = {}
@tractor.context
-async def open_trade_dialog(
+async def trades_dialogue(
    ctx: tractor.Context,
    broker: str,
-    fqme: str|None = None,  # if empty, we only boot broker mode
+    fqsn: str,
-    loglevel: str = 'warning',
+    loglevel: str = None,
 ) -> None:
-    # enable piker.clearing console log for *this* `brokerd` subactor
+    tractor.log.get_console_log(loglevel)
    get_console_log(
        level=loglevel,
        name=__name__,
    )
-    symcache: SymbologyCache
+    async with (
-    async with open_symcache(get_brokermod(broker)) as symcache:
+        data.open_feed(
            [fqsn],
            loglevel=loglevel,
        ) as feed,
-        acnt: Account
+    ):
-        ledger: TransactionLedger
+        pp_msgs: list[BrokerdPosition] = []
-        with (
+        pos: Position
-
+        token: str  # f'{symbol}.{self.broker}'
-            # TODO: probably do the symcache and ledger loading
+        for token, pos in _positions.items():
-            # implicitly behind this? Deliver an account, and ledger
+            pp_msgs.append(BrokerdPosition(
-            # pair or make the ledger an attr of the account?
+                broker=broker,
-            open_account(
+                account='paper',
-                broker,
+                symbol=pos.symbol.front_fqsn(),
-                'paper',
+                size=pos.size,
-                write_on_exit=True,
+                avg_price=pos.ppu,
            ) as acnt,
            open_trade_ledger(
                broker,
                'paper',
                symcache=symcache,
            ) as ledger
        ):
            # NOTE: WE MUST retreive market(pair) info from each
            # backend broker since ledger entries (in their
            # provider-native format) often don't contain necessary
            # market info per trade record entry..
            # FURTHER, if no fqme was passed in, we presume we're
            # running in "ledger-sync-only mode" and thus we load
            # mkt info for each symbol found in the ledger to
            # an acnt table manually.
            # TODO: how to process ledger info from backends?
            # - should we be rolling our own actor-cached version of these
            #   client API refs or using portal IPC to send requests to the
            #   existing brokerd daemon?
            # - alternatively we can possibly expect and use
            #   a `.broker.ledger.norm_trade()` ep?
            brokermod: ModuleType = get_brokermod(broker)
            gmi: Callable = getattr(brokermod, 'get_mkt_info', None)
            # update all transactions with mkt info before
            # loading any pps
            mkt_by_fqme: dict[str, MktPair] = {}
            if (
                fqme
                and fqme not in symcache.mktmaps
            ):
                log.warning(
                    f'Symcache for {broker} has no `{fqme}` entry?\n'
                    'Manually requesting mkt map data via `.get_mkt_info()`..'
                )
                bs_fqme, _, broker = fqme.rpartition('.')
                mkt, pair = await gmi(bs_fqme)
                mkt_by_fqme[mkt.fqme] = mkt
            # for each sym in the ledger load its `MktPair` info
            for tid, txdict in ledger.data.items():
                l_fqme: str = txdict.get('fqme') or txdict['fqsn']
                if (
                    gmi
                    and l_fqme not in symcache.mktmaps
                    and l_fqme not in mkt_by_fqme
                ):
                    log.warning(
                        f'Symcache for {broker} has no `{l_fqme}` entry?\n'
                        'Manually requesting mkt map data via `.get_mkt_info()`..'
                    )
                    mkt, pair = await gmi(
                        l_fqme.rstrip(f'.{broker}'),
                    )
                    mkt_by_fqme[l_fqme] = mkt
                # if an ``fqme: str`` input was provided we only
                # need a ``MktPair`` for that one market, since we're
                # running in real simulated-clearing mode, not just ledger
                # syncing.
                if (
                    fqme is not None
                    and fqme in mkt_by_fqme
                ):
                    break
            # update pos table from ledger history and provide a ``MktPair``
            # lookup for internal position accounting calcs.
            acnt.update_from_ledger(
                ledger,
                # NOTE: if the symcache fails on fqme lookup
                # (either sycache not yet supported or not filled
                # in) use manually constructed table from calling
                # the `.get_mkt_info()` provider EP above.
                _mktmap_table=mkt_by_fqme,
                only_require=list(mkt_by_fqme),
            )
            pp_msgs: list[BrokerdPosition] = []
            pos: Position
            token: str  # f'{symbol}.{self.broker}'
            for token, pos in acnt.pps.items():
                pp_msgs.append(BrokerdPosition(
                    broker=broker,
                    account='paper',
                    symbol=pos.mkt.fqme,
                    size=pos.cumsize,
                    avg_price=pos.ppu,
                ))
            await ctx.started((
                pp_msgs,
                ['paper'],
            ))
-            # write new positions state in case ledger was
+        # TODO: load paper positions per broker from .toml config file
-            # newer then that tracked in pps.toml
+        # and pass as symbol to position data mapping: ``dict[str, dict]``
-            acnt.write_config()
+        await ctx.started((
            pp_msgs,
            ['paper'],
        ))
-            # exit early since no fqme was passed,
+        async with (
-            # normally this case is just to load
+            ctx.open_stream() as ems_stream,
-            # positions "offline".
+            trio.open_nursery() as n,
-            if fqme is None:
+        ):
-                log.warning(
+            client = PaperBoi(
-                    'Paper engine only running in position delivery mode!\n'
+                broker,
-                    'NO SIMULATED CLEARING LOOP IS ACTIVE!'
+                ems_stream,
-                )
+                _buys=_buys,
-                await trio.sleep_forever()
+                _sells=_sells,
                return
-            feed: Feed
+                _reqids=_reqids,
            async with (
                open_feed(
                    [fqme],
                    loglevel=loglevel,
                ) as feed,
            ):
                # sanity check all the mkt infos
                for fqme, flume in feed.flumes.items():
                    mkt: MktPair = symcache.mktmaps.get(fqme) or mkt_by_fqme[fqme]
                    if mkt != flume.mkt:
                        diff: tuple = mkt - flume.mkt
                        log.warning(
                            'MktPair sig mismatch?\n'
                            f'{pformat(diff)}'
                        )
-                get_cost: Callable = getattr(
+                # TODO: load paper positions from ``positions.toml``
-                    brokermod,
+                _positions=_positions,
                    'get_cost',
                    None,
                )
-                async with (
+                # TODO: load postions from ledger file
-                    ctx.open_stream() as ems_stream,
+                _trade_ledger={},
-                    trio.open_nursery() as n,
+            )
                ):
                    client = PaperBoi(
                        broker=broker,
                        ems_trades_stream=ems_stream,
                        acnt=acnt,
                        ledger=ledger,
                        fees=get_cost,
-                        _buys=_buys,
+            n.start_soon(
-                        _sells=_sells,
+                handle_order_requests,
-                        _reqids=_reqids,
+                client,
                ems_stream,
            )
-                        _mkts=mkt_by_fqme,
+            # paper engine simulator clearing task
-
+            await simulate_fills(feed.streams[broker], client)
                    )
                    n.start_soon(
                        handle_order_requests,
                        client,
                        ems_stream,
                    )
                    # paper engine simulator clearing task
                    await simulate_fills(feed.streams[broker], client)
-@acm
+@asynccontextmanager
 async def open_paperboi(
-    fqme: str | None = None,
+    fqsn: str,
-    broker: str | None = None,
+    loglevel: str,
    loglevel: str | None = None,
 ) -> Callable:
    '''
@ -759,91 +592,28 @@ async def open_paperboi(
    its context.
    '''
-    if not fqme:
+    broker, symbol, expiry = unpack_fqsn(fqsn)
        assert broker, 'One of `broker` or `fqme` is required siss..!'
    else:
        broker, _, _, _ = unpack_fqme(fqme)
    we_spawned: bool = False
    service_name = f'paperboi.{broker}'
    async with (
-        find_service(service_name) as portal,
+        tractor.find_actor(service_name) as portal,
-        tractor.open_nursery() as an,
+        tractor.open_nursery() as tn,
    ):
-        # NOTE: only spawn if no paperboi already is up since we likely
+        # only spawn if no paperboi already is up
-        # don't need more then one actor for simulated order clearing
+        # (we likely don't need more then one proc for basic
-        # per broker-backend.
+        # simulated order clearing)
        if portal is None:
            log.info('Starting new paper-engine actor')
-            portal = await an.start_actor(
+            portal = await tn.start_actor(
                service_name,
                enable_modules=[__name__]
            )
            we_spawned = True
        async with portal.open_context(
-            open_trade_dialog,
+            trades_dialogue,
            broker=broker,
-            fqme=fqme,
+            fqsn=fqsn,
            loglevel=loglevel,
        ) as (ctx, first):
            yield ctx, first
            # ALWAYS tear down connection AND any newly spawned
            # paperboi actor on exit!
            await ctx.cancel()
            if we_spawned:
                await portal.cancel_actor()
 def norm_trade(
    tid: str,
    txdict: dict,
    pairs: dict[str, Struct],
    symcache: SymbologyCache | None = None,
    brokermod: ModuleType | None = None,
 ) -> Transaction:
    from pendulum import (
        DateTime,
        parse,
    )
    # special field handling for datetimes
    # to ensure pendulum is used!
    dt: DateTime = parse(txdict['dt'])
    expiry: str | None = txdict.get('expiry')
    fqme: str = txdict.get('fqme') or txdict.pop('fqsn')
    price: float = txdict['price']
    size: float =  txdict['size']
    cost: float = txdict.get('cost', 0)
    if (
        brokermod
        and (get_cost := getattr(
            brokermod,
            'get_cost',
            False,
        ))
    ):
        cost = get_cost(
            price,
            size,
            is_taker=True,
        )
    return Transaction(
        fqme=fqme,
        tid=txdict['tid'],
        dt=dt,
        price=price,
        size=size,
        cost=cost,
        bs_mktid=txdict['bs_mktid'],
        expiry=parse(expiry) if expiry else None,
        etype='clear',
    )
--- a/piker/clearing/_util.py
+++ b/piker/clearing/_util.py
@ -1,96 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 Sub-sys module commons.
 """
 from collections import ChainMap
 from functools import partial
 from typing import Any
 from ..log import (
    get_logger,
    get_console_log,
 )
 from piker.types import Struct
 subsys: str = 'piker.clearing'
 log = get_logger(
    name='piker.clearing',
 )
 # TODO, oof doesn't this ignore the `loglevel` then???
 get_console_log = partial(
    get_console_log,
    name='clearing',
 )
 class OrderDialogs(Struct):
    '''
    Order control dialog (and thus transaction) tracking via
    message recording.
    Allows easily recording messages associated with a given set of
    order control transactions and looking up the latest field
    state using the entire (reverse chronological) msg flow.
    '''
    _flows: dict[str, ChainMap] = {}
    def add_msg(
        self,
        oid: str,
        msg: dict,
    ) -> None:
        # NOTE: manually enter a new map on the first msg add to
        # avoid creating one with an empty dict first entry in
        # `ChainMap.maps` which is the default if none passed at
        # init.
        cm: ChainMap = self._flows.get(oid)
        if cm:
            cm.maps.insert(0, msg)
        else:
            cm = ChainMap(msg)
            self._flows[oid] = cm
    # TODO: wrap all this in the `collections.abc.Mapping` interface?
    def get(
        self,
        oid: str,
    ) -> ChainMap[str, Any]:
        '''
        Return the dialog `ChainMap` for provided id.
        '''
        return self._flows.get(oid, None)
    def pop(
        self,
        oid: str,
    ) -> ChainMap[str, Any]:
        '''
        Pop and thus remove the `ChainMap` containing the msg flow
        for the given order id.
        '''
        if (flow := self._flows.pop(oid, None)) is None:
            log.warning(f'No flow found for oid: {oid}')
        return flow
--- a/piker/cli/init.py
+++ b/piker/cli/init.py
@ -1,192 +1,113 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present Tyler Goodlet
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
 # (in stewardship for pikers, everywhere.)
-# This program is free software: you can redistribute it and/or
+# This program is free software: you can redistribute it and/or modify
-# modify it under the terms of the GNU Affero General Public
+# it under the terms of the GNU Affero General Public License as published by
-# License as published by the Free Software Foundation, either
+# the Free Software Foundation, either version 3 of the License, or
-# version 3 of the License, or (at your option) any later version.
+# (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# Affero General Public License for more details.
+# GNU Affero General Public License for more details.
-# You should have received a copy of the GNU Affero General Public
+# You should have received a copy of the GNU Affero General Public License
-# License along with this program.  If not, see
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
 # <https://www.gnu.org/licenses/>.
 '''
 CLI commons.
 '''
 import os
-# from contextlib import AsyncExitStack
+from pprint import pformat
 from types import ModuleType
 import click
 import trio
 import tractor
 from tractor._multiaddr import parse_maddr
-from ..log import (
+from ..log import get_console_log, get_logger, colorize_json
    get_console_log,
    get_logger,
    colorize_json,
 )
 from ..brokers import get_brokermod
-from ..service import (
+from .._daemon import (
    _default_registry_host,
    _default_registry_port,
 )
 from .. import config
-log = get_logger('piker.cli')
+log = get_logger('cli')
 def load_trans_eps(
    network: dict | None = None,
    maddrs: list[tuple] | None = None,
 ) -> dict[str, dict[str, dict]]:
    # transport-oriented endpoint multi-addresses
    eps: dict[
        str,  # service name, eg. `pikerd`, `emsd`..
        # libp2p style multi-addresses parsed into prot layers
        list[dict[str, str | int]]
    ] = {}
    if (
        network
        and
        not maddrs
    ):
        # load network section and (attempt to) connect all endpoints
        # which are reachable B)
        for key, maddrs in network.items():
            match key:
                # TODO: resolve table across multiple discov
                # prots Bo
                case 'resolv':
                    pass
                case 'pikerd':
                    dname: str = key
                    for maddr in maddrs:
                        layers: dict = parse_maddr(maddr)
                        eps.setdefault(
                            dname,
                            [],
                        ).append(layers)
    elif maddrs:
        # presume user is manually specifying the root actor ep.
        eps['pikerd'] = [parse_maddr(maddr)]
    return eps
@click.command()
@click.option('--loglevel', '-l', default='warning', help='Logging level')
@click.option('--tl', is_flag=True, help='Enable tractor logging')
@click.option('--pdb', is_flag=True, help='Enable tractor debug mode')
@click.option('--host', '-h', default=None, help='Host addr to bind')
@click.option('--port', '-p', default=None, help='Port number to bind')
@click.option(
-    '--loglevel',
+    '--tsdb',
    '-l',
    default='warning',
    help='Logging level',
 )
@click.option(
    '--tl',
    is_flag=True,
-    help='Enable tractor-runtime logs',
+    help='Enable local ``marketstore`` instance'
 )
@click.option(
    '--pdb',
    is_flag=True,
    help='Enable tractor debug mode',
 )
@click.option(
    '--maddr',
    '-m',
    default=None,
    help='Multiaddrs to bind or contact',
 )
 def pikerd(
    maddr: list[str] | None,
    loglevel: str,
    host: str,
    port: int,
    tl: bool,
    pdb: bool,
    tsdb: bool,
 ):
    '''
-    Start the "root service actor", `pikerd`, run it until
+    Spawn the piker broker-daemon.
    cancellation.
    This "root daemon" operates as the top most service-mngr and
    subsys-as-subactor supervisor, think of it as the "init proc" of
    any of any `piker` application or daemon-process tree.
    '''
-    # from tractor.devx import maybe_open_crash_handler
+    from .._daemon import open_pikerd
-    # with maybe_open_crash_handler(pdb=False):
+    log = get_console_log(loglevel)
    log = get_console_log(
        level=loglevel,
        with_tractor_log=tl,
    )
    if pdb:
        log.warning((
            "\n"
-            "!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!\n"
+            "!!! You have enabled daemon DEBUG mode !!!\n"
-            "When a `piker` daemon crashes it will block the "
+            "If a daemon crashes it will likely block"
-            "task-thread until resumed from console!\n"
+            " the service until resumed from console!\n"
            "\n"
        ))
-    # service-actor registry endpoint socket-address set
+    reg_addr: None | tuple[str, int] = None
-    regaddrs: list[tuple[str, int]] = []
+    if host or port:
-
+        reg_addr = (
-    conf, _ = config.load(
+            host or _default_registry_host,
-        conf_name='conf',
+            int(port) or _default_registry_port,
    )
    network: dict = conf.get('network')
    if (
        network is None
        and not maddr
    ):
        regaddrs = [(
            _default_registry_host,
            _default_registry_port,
        )]
    else:
        eps: dict = load_trans_eps(
            network,
            maddr,
        )
        for layers in eps['pikerd']:
            regaddrs.append((
                layers['ipv4']['addr'],
                layers['tcp']['port'],
            ))
    from .. import service
    async def main():
-        service_mngr: service.Services
+
        async with (
-            service.open_pikerd(
+            open_pikerd(
                registry_addrs=regaddrs,
                loglevel=loglevel,
                debug_mode=pdb,
-                # enable_transports=['uds'],
+                registry_addr=reg_addr,
-                enable_transports=['tcp'],
+
-            ) as service_mngr,
+            ),  # normally delivers a ``Services`` handle
            trio.open_nursery() as n,
        ):
-            assert service_mngr
+            if tsdb:
-            # ?TODO? spawn all other sub-actor daemons according to
+                from piker.data._ahab import start_ahab
-            # multiaddress endpoint spec defined by user config
+                from piker.data.marketstore import start_marketstore
                log.info('Spawning `marketstore` supervisor')
                ctn_ready, config, (cid, pid) = await n.start(
                    start_ahab,
                    'marketstored',
                    start_marketstore,
                )
                log.info(
                    f'`marketstored` up!\n'
                    f'pid: {pid}\n'
                    f'container id: {cid[:12]}\n'
                    f'config: {pformat(config)}'
                )
            await trio.sleep_forever()
    trio.run(main)
@ -202,24 +123,8 @@ def pikerd(
@click.option('--loglevel', '-l', default='warning', help='Logging level')
@click.option('--tl', is_flag=True, help='Enable tractor logging')
@click.option('--configdir', '-c', help='Configuration directory')
-@click.option(
+@click.option('--host', '-h', default=None, help='Host addr to bind')
-    '--pdb',
+@click.option('--port', '-p', default=None, help='Port number to bind')
    is_flag=True,
    help='Enable runtime debug mode ',
 )
@click.option(
    '--maddr',
    '-m',
    default=None,
    multiple=True,
    help='Multiaddr to bind',
 )
@click.option(
    '--regaddr',
    '-r',
    default=None,
    help='Registrar addr to contact',
 )
@click.pass_context
 def cli(
    ctx: click.Context,
@ -227,27 +132,14 @@ def cli(
    loglevel: str,
    tl: bool,
    configdir: str,
-    pdb: bool,
+    host: str,
-
+    port: int,
    # TODO: make these list[str] with multiple -m maddr0 -m maddr1
    maddr: list[str],
    regaddr: str,
 ) -> None:
    '''
    The "root" `piker`-cmd CLI endpoint.
    NOTE, this def generally relies on and requires a sub-cmd to be
    provided by the user, OW only a `--help` msg (listing said
    subcmds) will be dumped to console.
    '''
    if configdir is not None:
        assert os.path.isdir(configdir), f"`{configdir}` is not a valid path"
        config._override_config_dir(configdir)
    # TODO: for typer see
    # https://typer.tiangolo.com/tutorial/commands/context/
    ctx.ensure_object(dict)
    if not brokers:
@ -255,25 +147,15 @@ def cli(
        from piker.brokers import __brokers__
        brokers = __brokers__
-    brokermods: dict[str, ModuleType] = {
+    brokermods = [get_brokermod(broker) for broker in brokers]
        broker: get_brokermod(broker) for broker in brokers
    }
    assert brokermods
-    # TODO: load endpoints from `conf::[network].pikerd`
+    reg_addr: None | tuple[str, int] = None
-    # - pikerd vs. regd, separate registry daemon?
+    if host or port:
-    # - expose datad vs. brokerd?
+        reg_addr = (
-    # - bind emsd with certain perms on public iface?
+            host or _default_registry_host,
-    regaddrs: list[tuple[str, int]] = regaddr or [(
+            int(port) or _default_registry_port,
-        _default_registry_host,
+        )
        _default_registry_port,
    )]
    # TODO: factor [network] section parsing out from pikerd
    # above and call it here as well.
    # if maddr:
    #     for addr in maddr:
    #         layers: dict = parse_maddr(addr)
    ctx.obj.update({
        'brokers': brokers,
@ -283,12 +165,7 @@ def cli(
        'log': get_console_log(loglevel),
        'confdir': config._config_dir,
        'wl_path': config._watchlists_data_path,
-        'registry_addrs': regaddrs,
+        'registry_addr': reg_addr,
        'pdb': pdb,  # debug mode flag
        # TODO: endpoint parsing, pinging and binding
        # on no existing server.
        # 'maddrs': maddr,
    })
    # allow enabling same loglevel in ``tractor`` machinery
@ -300,101 +177,47 @@ def cli(
@click.option('--tl', is_flag=True, help='Enable tractor logging')
@click.argument('ports', nargs=-1, required=False)
@click.pass_obj
-def services(
+def services(config, tl, ports):
    config,
    tl: bool,
    ports: list[int],
 ):
    '''
    List all `piker` "service deamons" to the console in
    a `json`-table which maps each actor's UID in the form,
-    `{service_name}.{subservice_name}.{UUID}`
+    from .._daemon import (
    to its (primary) IPC server address.
    (^TODO, should be its multiaddr form once we support it)
    Note that by convention actors which operate as "headless"
    processes (those without GUIs/graphics, and which generally
    parent some noteworthy subsystem) are normally suffixed by
    a "d" such as,
    - pikerd: the root runtime supervisor
    - brokerd: a broker-backend order ctl daemon
    - emsd: the internal dark-clearing and order routing daemon
    - datad: a data-provider-backend data feed daemon
    - samplerd: the real-time data sampling and clock-syncing daemon
    "Headed units" are normally just given an obvious app-like name
    with subactors indexed by `.` such as,
    - chart: the primary modal charting iface, a Qt app
    - chart.fsp_0: a financial-sig-proc cascade instance which
                   delivers graphics to a parent `chart` app.
    - polars_boi: some (presumably) `polars` using console app.
    '''
    from piker.service import (
        open_piker_runtime,
        _default_registry_port,
        _default_registry_host,
    )
-    # !TODO, mk this to work with UDS!
+    host = _default_registry_host
    host: str = _default_registry_host
    if not ports:
-        ports: list[int] = [_default_registry_port]
+        ports = [_default_registry_port]
    addr = tractor._addr.wrap_address(
        addr=(host, ports[0])
    )
    async def list_services():
        nonlocal host
        async with (
            open_piker_runtime(
                name='service_query',
-                loglevel=(
+                loglevel=config['loglevel'] if tl else None,
                    config['loglevel']
                    if tl
                    else None
                ),
            ),
-            tractor.get_registry(
+            tractor.get_arbiter(
-                addr=addr,
+                host=host,
                port=ports[0]
            ) as portal
        ):
-            registry = await portal.run_from_ns(
+            registry = await portal.run_from_ns('self', 'get_registry')
                'self',
                'get_registry',
            )
            json_d = {}
            for key, socket in registry.items():
-                json_d[key] = f'{socket}'
+                host, port = socket
-
+                json_d[key] = f'{host}:{port}'
            click.echo(f"{colorize_json(json_d)}")
    trio.run(list_services)
 def _load_clis() -> None:
-    '''
+    from ..data import marketstore  # noqa
-    Dynamically load and register all subsys CLI endpoints (at call
+    from ..data import cli  # noqa
    time).
    NOTE, obviously this is normally expected to be called at
    `import` time and implicitly relies on our use of various
    `click`/`typer` decorator APIs.
    '''
    from ..brokers import cli  # noqa
    from ..ui import cli  # noqa
    from ..watchlists import cli  # noqa
    # typer implemented
    from ..storage import cli  # noqa
    from ..accounting import cli  # noqa
-
+# load downstream cli modules
 # load all subsytem cli eps
 _load_clis()
--- a/piker/config.py
+++ b/piker/config.py
@ -15,143 +15,109 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
-Platform configuration (files) mgmt.
+Broker configuration mgmt.
 """
 import platform
 import sys
 import os
 from os import path
 from os.path import dirname
 import shutil
-from typing import (
+from typing import Optional
    Callable,
    MutableMapping,
 )
 from pathlib import Path
 from bidict import bidict
-import platformdirs
+import toml
 import tomlkit
 try:
    import tomllib
 except ModuleNotFoundError:
    import tomli as tomllib
 from .log import get_logger
 log = get_logger('broker-config')
-# XXX NOTE: orig impl was taken from `click`
+# taken from ``click`` since apparently they have some
-# |_https://github.com/pallets/click/blob/main/src/click/utils.py#L449
+# super weirdness with sigint and sudo..no clue
-#
+def get_app_dir(app_name, roaming=True, force_posix=False):
-# (since apparently they have some super weirdness with SIGINT and
+    r"""Returns the config folder for the application.  The default behavior
 # sudo.. no clue we're probably going to slowly just modify it to our
 # own version over time..)
 #
 def get_app_dir(
    app_name: str,
    roaming: bool = True,
    force_posix: bool = False,
 ) -> str:
    '''
    Returns the config folder for the application.  The default behavior
    is to return whatever is most appropriate for the operating system.
-    ----
+    To give you an idea, for an app called ``"Foo Bar"``, something like
-    NOTE, below is originally from `click` impl fn, we can prolly remove?
+    the following folders could be returned:
    ----
    Mac OS X:
      ``~/Library/Application Support/Foo Bar``
    Mac OS X (POSIX):
      ``~/.foo-bar``
    Unix:
      ``~/.config/foo-bar``
    Unix (POSIX):
      ``~/.foo-bar``
    Win XP (roaming):
      ``C:\Documents and Settings\<user>\Local Settings\Application Data\Foo``
    Win XP (not roaming):
      ``C:\Documents and Settings\<user>\Application Data\Foo Bar``
    Win 7 (roaming):
      ``C:\Users\<user>\AppData\Roaming\Foo Bar``
    Win 7 (not roaming):
      ``C:\Users\<user>\AppData\Local\Foo Bar``
    .. versionadded:: 2.0
    :param app_name: the application name.  This should be properly capitalized
                     and can contain whitespace.
    :param roaming: controls if the folder should be roaming or not on Windows.
                    Has no affect otherwise.
    :param force_posix: if this is set to `True` then on any POSIX system the
                        folder will be stored in the home folder with a leading
                        dot instead of the XDG config home or darwin's
                        application support folder.
-    '''
+    """
    # NOTE: for testing with `pytest` we leverage the `tmp_dir`
    # fixture to generate (and clean up) a test-request-specific
    # directory for isolated configuration files such that,
    # - multiple tests can run (possibly in parallel) without data races
    #   on the config state,
    # - we don't need to ever worry about leaking configs into the
    #   system thus avoiding needing to manage config cleaup fixtures or
    #   other bothers (since obviously `tmp_dir` cleans up after itself).
    #
    # In order to "pass down" the test dir path to all (sub-)actors in
    # the actor tree we preload the root actor's runtime vars state (an
    # internal mechanism for inheriting state down an actor tree in
    # `tractor`) with the testing dir and check for it whenever we
    # detect `pytest` is being used (which it isn't under normal
    # operation).
    # if "pytest" in sys.modules:
    #     import tractor
    #     actor = tractor.current_actor(err_on_no_runtime=False)
    #     if actor:  # runtime is up
    #         rvs = tractor._state._runtime_vars
    #         import pdbp; pdbp.set_trace()
    #         testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
    #         assert testdirpath.exists(), 'piker test harness might be borked!?'
    #         app_name = str(testdirpath)
-    os_name: str = platform.system()
+    def _posixify(name):
-    conf_dir: Path = platformdirs.user_config_path()
+        return "-".join(name.split()).lower()
    app_dir: Path = conf_dir / app_name
-    # ?TODO, from `click`; can remove?
+    # if WIN:
    if platform.system() == 'Windows':
        key = "APPDATA" if roaming else "LOCALAPPDATA"
        folder = os.environ.get(key)
        if folder is None:
            folder = os.path.expanduser("~")
        return os.path.join(folder, app_name)
    if force_posix:
        def _posixify(name):
            return "-".join(name.split()).lower()
        return os.path.join(
-            os.path.expanduser(
+            os.path.expanduser("~/.{}".format(_posixify(app_name))))
-                "~/.{}".format(
+    if sys.platform == "darwin":
-                    _posixify(app_name)
+        return os.path.join(
-                )
+            os.path.expanduser("~/Library/Application Support"), app_name
            )
        )
-
+    return os.path.join(
-    log.info(
+        os.environ.get("XDG_CONFIG_HOME", os.path.expanduser("~/.config")),
-        f'Using user config directory,\n'
+        _posixify(app_name),
        f'platform.system(): {os_name!r}\n'
        f'conf_dir: {conf_dir!r}\n'
        f'app_dir: {conf_dir!r}\n'
    )
    return app_dir
-_click_config_dir: Path = Path(get_app_dir('piker'))
+_config_dir = _click_config_dir = get_app_dir('piker')
-_config_dir: Path = _click_config_dir
+_parent_user = os.environ.get('SUDO_USER')
-# NOTE: when using `sudo` we attempt to determine the non-root user
+if _parent_user:
-# and still use their normal config dir.
+    non_root_user_dir = os.path.expanduser(
-if (
+        f'~{_parent_user}'
    (_parent_user := os.environ.get('SUDO_USER'))
    and
    _parent_user != 'root'
 ):
    non_root_user_dir = Path(
        os.path.expanduser(f'~{_parent_user}')
    )
-    root: str = 'root'
+    root = 'root'
    _ccds: str = str(_click_config_dir)  # click config dir as string
    i_tail: int = int(_ccds.rfind(root) + len(root))
    _config_dir = (
-        non_root_user_dir
+        non_root_user_dir +
-        /
+        _click_config_dir[
-        Path(_ccds[i_tail+1:])  # +1 to capture trailing '/'
+            _click_config_dir.rfind(root) + len(root):
        ]
    )
 _conf_names: set[str] = {
-    'conf',  # god config
+    'brokers',
-    'brokers',  # sec backend deatz
+    'pps',
-    'watchlists',  # (user defined) market lists
+    'trades',
    'watchlists',
 }
-# TODO: probably drop all this super legacy, questrade specific,
+_watchlists_data_path = os.path.join(_config_dir, 'watchlists.json')
 # config stuff XD ?
 _watchlists_data_path: Path = _config_dir / Path('watchlists.json')
 _context_defaults = dict(
    default_map={
        # Questrade specific quote poll rates
@ -165,14 +131,6 @@ _context_defaults = dict(
 )
 class ConfigurationError(Exception):
    'Misconfigured settings, likely in a TOML file.'
 class NoSignature(ConfigurationError):
    'No credentials setup for broker backend!'
 def _override_config_dir(
    path: str
 ) -> None:
@ -187,19 +145,10 @@ def _conf_fn_w_ext(
    return f'{name}.toml'
 def get_conf_dir() -> Path:
    '''
    Return the user configuration directory ``Path``
    on the local filesystem.
    '''
    return _config_dir
 def get_conf_path(
    conf_name: str = 'brokers',
-) -> Path:
+) -> str:
    '''
    Return the top-level default config path normally under
    ``~/.config/piker`` on linux for a given ``conf_name``, the config
@ -207,6 +156,7 @@ def get_conf_path(
    Contains files such as:
    - brokers.toml
    - pp.toml
    - watchlists.toml
    # maybe coming soon ;)
@ -214,105 +164,67 @@ def get_conf_path(
    - strats.toml
    '''
-    if 'account.' not in conf_name:
+    assert conf_name in _conf_names
        assert str(conf_name) in _conf_names
    fn = _conf_fn_w_ext(conf_name)
-    return _config_dir / Path(fn)
+    return os.path.join(
-
+        _config_dir,
-
+        fn,
 def repodir() -> Path:
    '''
    Return the abspath as ``Path`` to the git repo's root dir.
    '''
    repodir: Path = Path(__file__).absolute().parent.parent
    confdir: Path = repodir / 'config'
    if not confdir.is_dir():
        # prolly inside stupid GH actions CI..
        repodir: Path = Path(os.environ.get('GITHUB_WORKSPACE'))
        confdir: Path = repodir / 'config'
    assert confdir.is_dir(), (
        f'{confdir} DNE, {repodir} is likely incorrect!'
    )
-    return repodir
+
 def repodir():
    '''
    Return the abspath to the repo directory.
    '''
    dirpath = path.abspath(
        # we're 3 levels down in **this** module file
        dirname(dirname(os.path.realpath(__file__)))
    )
    return dirpath
 def load(
-    # NOTE: always appended with .toml suffix
+    conf_name: str = 'brokers',
-    conf_name: str = 'conf',
+    path: str = None,
    path: Path | None = None,
    decode: Callable[
        [str | bytes,],
        MutableMapping,
    ] = tomllib.loads,
    touch_if_dne: bool = True,
    **tomlkws,
-) -> tuple[dict, Path]:
+) -> (dict, str):
    '''
    Load config file by name.
    If desired config is not in the top level piker-user config path then
    pass the `path: Path` explicitly.
    '''
-    # create the $HOME/.config/piker dir if dne
+    path = path or get_conf_path(conf_name)
-    if not _config_dir.is_dir():
+
-        _config_dir.mkdir(
+    if not os.path.isdir(_config_dir):
-            parents=True,
+        os.mkdir(_config_dir)
-            exist_ok=True,
+
-        )
+    if not os.path.isfile(path):
-
+        fn = _conf_fn_w_ext(conf_name)
-    path_provided: bool = path is not None
+
-    path: Path = path or get_conf_path(conf_name)
+        template = os.path.join(
-
+            repodir(),
-    if (
+            'config',
-        not path.is_file()
+            fn
        and
        touch_if_dne
    ):
        # only do a template if no path provided,
        # just touch an empty file with same name.
        if path_provided:
            with path.open(mode='x'):
                pass
        # try to copy in a template config to the user's dir if one
        # exists.
        else:
            fn: str = _conf_fn_w_ext(conf_name)
            template: Path = repodir() / 'config' / fn
            if template.is_file():
                shutil.copyfile(template, path)
            elif fn and template:
                assert template.is_file(), f'{template} is not a file!?'
            assert path.is_file(), f'Config file {path} not created!?'
    with path.open(mode='r') as fp:
        config: dict = decode(
            fp.read(),
            **tomlkws,
        )
        # try to copy in a template config to the user's directory
        # if one exists.
        if os.path.isfile(template):
            shutil.copyfile(template, path)
    else:
        with open(path, 'r'):
            pass  # touch it
    config = toml.load(path, **tomlkws)
    log.debug(f"Read config file {path}")
    return config, path
 def write(
    config: dict,  # toml config as dict
-
+    name: str = 'brokers',
-    name: str | None = None,
+    path: str = None,
    path: Path | None = None,
    fail_empty: bool = True,
    **toml_kwargs,
 ) -> None:
@ -322,41 +234,34 @@ def write(
    Create a ``brokers.ini`` file if one does not exist.
    '''
-    if name:
+    path = path or get_conf_path(name)
-        path: Path = path or get_conf_path(name)
+    dirname = os.path.dirname(path)
-        dirname: Path = path.parent
+    if not os.path.isdir(dirname):
-        if not dirname.is_dir():
+        log.debug(f"Creating config dir {_config_dir}")
-            log.debug(f"Creating config dir {_config_dir}")
+        os.makedirs(dirname)
            dirname.mkdir()
-    if (
+    if not config:
        not config
        and fail_empty
    ):
        raise ValueError(
-            "Watch out you're trying to write a blank config!"
+            "Watch out you're trying to write a blank config!")
        )
    log.debug(
        f"Writing config `{name}` file to:\n"
        f"{path}"
    )
-    with path.open(mode='w') as fp:
+    with open(path, 'w') as cf:
-        return tomlkit.dump(  # preserve style on write B)
+        return toml.dump(
            config,
-            fp,
+            cf,
            **toml_kwargs,
        )
 def load_accounts(
-    providers: list[str] | None = None
+    providers: Optional[list[str]] = None
-) -> bidict[str, str | None]:
+) -> bidict[str, Optional[str]]:
-    conf, path = load(
+    conf, path = load()
        conf_name='brokers',
    )
    accounts = bidict()
    for provider_name, section in conf.items():
        accounts_section = section.get('accounts')
--- a/piker/data/init.py
+++ b/piker/data/init.py
@ -22,7 +22,13 @@ and storing data from your brokers as well as
 sharing live streams over a network.
 """
-from .ticktools import iterticks
+import tractor
 import trio
 from ..log import (
    get_console_log,
 )
 from ._normalize import iterticks
 from ._sharedmem import (
    maybe_open_shm_array,
    attach_shm_array,
@ -30,42 +36,53 @@ from ._sharedmem import (
    get_shm_token,
    ShmArray,
 )
 from ._source import (
    def_iohlcv_fields,
    def_ohlcv_fields,
 )
 from .feed import (
    Feed,
    open_feed,
 )
 from .flows import Flume
 from ._symcache import (
    SymbologyCache,
    open_symcache,
    get_symcache,
    match_from_pairs,
 )
 from ._sampling import open_sample_stream
 from ..types import Struct
-__all__: list[str] = [
+__all__ = [
    'Flume',
    'Feed',
    'open_feed',
    'ShmArray',
    'iterticks',
    'maybe_open_shm_array',
    'match_from_pairs',
    'attach_shm_array',
    'open_shm_array',
    'get_shm_token',
    'def_iohlcv_fields',
    'def_ohlcv_fields',
    'open_symcache',
    'open_sample_stream',
    'get_symcache',
    'Struct',
    'SymbologyCache',
    'types',
 ]
@tractor.context
 async def _setup_persistent_brokerd(
    ctx: tractor.Context,
    brokername: str,
 ) -> None:
    '''
    Allocate a actor-wide service nursery in ``brokerd``
    such that feeds can be run in the background persistently by
    the broker backend as needed.
    '''
    get_console_log(tractor.current_actor().loglevel)
    from .feed import (
        _bus,
        get_feed_bus,
    )
    global _bus
    assert not _bus
    async with trio.open_nursery() as service_nursery:
        # assign a nursery to the feeds bus for spawning
        # background tasks from clients
        get_feed_bus(brokername, service_nursery)
        # unblock caller
        await ctx.started()
        # we pin this task to keep the feeds manager active until the
        # parent actor decides to tear it down
        await trio.sleep_forever()
--- a/piker/service/_ahab.py
+++ b/piker/service/_ahab.py
@ -15,13 +15,9 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
-Supervisor for ``docker`` with included async and SC wrapping to
+Supervisor for docker with included specific-image service helpers.
 ensure a cancellable container lifetime system.
 '''
 from __future__ import annotations
 from collections import ChainMap
 from functools import partial
 import os
 import time
 from typing import (
@ -49,14 +45,10 @@ from requests.exceptions import (
    ReadTimeout,
 )
-from piker.log import (
+from ..log import get_logger, get_console_log
    get_console_log,
    get_logger,
 )
 from ._mngr import Services
 from .. import config
-log = get_logger(name=__name__)
+log = get_logger(__name__)
 class DockerNotStarted(Exception):
@ -132,19 +124,8 @@ class Container:
    async def process_logs_until(
        self,
-        log_msg_key: str,
+        patt: str,
-
+        bp_on_msg: bool = False,
        # this is a predicate func for matching log msgs emitted by the
        # underlying containerized app
        patt_matcher: Callable[[str], bool],
        # XXX WARNING XXX: do not touch this sleep value unless
        # you know what you are doing! the value is critical to
        # making sure the caller code inside the startup context
        # does not timeout BEFORE we receive a match on the
        # ``patt_matcher()`` predicate above.
        checkpoint_period: float = 0.001,
    ) -> bool:
        '''
        Attempt to capture container log messages and relay through our
@ -155,15 +136,6 @@ class Container:
        while True:
            logs = self.cntr.logs()
            try:
                logs = self.cntr.logs()
            except (
                docker.errors.NotFound,
                docker.errors.APIError
            ):
                log.exception('Failed to parse logs?')
                return False
            entries = logs.decode().split('\n')
            for entry in entries:
@ -171,48 +143,34 @@ class Container:
                if not entry:
                    continue
                entry = entry.strip()
                try:
-                    record = json.loads(entry)
+                    record = json.loads(entry.strip())
                    msg = record[log_msg_key]
                    level = record['level']
                except json.JSONDecodeError:
-                    msg = entry
+                    if 'Error' in entry:
-                    level = 'error'
+                        raise RuntimeError(entry)
                    raise
-                # TODO: do we need a more general mechanism
+                msg = record['msg']
-                # for these kinda of "log record entries"?
+                level = record['level']
-                # if 'Error' in entry:
+                if msg and entry not in seen_so_far:
                #     raise RuntimeError(entry)
                if (
                    msg
                    and entry not in seen_so_far
                ):
                    seen_so_far.add(entry)
-                    getattr(
+                    if bp_on_msg:
-                        log,
+                        await tractor.breakpoint()
                        level.lower(),
                        log.error
                    )(f'{msg}')
-                    if level == 'fatal':
+                    getattr(log, level, log.error)(f'{msg}')
                    # print(f'level: {level}')
                    if level in ('error', 'fatal'):
                        raise ApplicationLogError(msg)
-                if await patt_matcher(msg):
+                if patt in msg:
                    return True
                # do a checkpoint so we don't block if cancelled B)
-                await trio.sleep(checkpoint_period)
+                await trio.sleep(0.01)
        return False
    @property
    def cuid(self) -> str:
        fqcn: str = self.cntr.attrs['Config']['Image']
        return f'{fqcn}[{self.cntr.short_id}]'
    def try_signal(
        self,
        signal: str = 'SIGINT',
@ -248,39 +206,28 @@ class Container:
    async def cancel(
        self,
-        log_msg_key: str,
+        stop_msg: str,
        stop_predicate: Callable[[str], bool],
        hard_kill: bool = False,
    ) -> None:
        '''
        Attempt to cancel this container gracefully, fail over to
        a hard kill on timeout.
        '''
        cid = self.cntr.id
        # first try a graceful cancel
        log.cancel(
-            f'SIGINT cancelling container: {self.cuid}\n'
+            f'SIGINT cancelling container: {cid}\n'
-            'waiting on stop predicate...'
+            f'waiting on stop msg: "{stop_msg}"'
        )
        self.try_signal('SIGINT')
        start = time.time()
        for _ in range(6):
-            with trio.move_on_after(1) as cs:
+            with trio.move_on_after(0.5) as cs:
-                log.cancel(
+                log.cancel('polling for CNTR logs...')
                    'polling for CNTR logs for {stop_predicate}..'
                )
                try:
-                    await self.process_logs_until(
+                    await self.process_logs_until(stop_msg)
                        log_msg_key,
                        stop_predicate,
                    )
                except ApplicationLogError:
                    hard_kill = True
                else:
@ -338,16 +285,11 @@ class Container:
 async def open_ahabd(
    ctx: tractor.Context,
    endpoint: str,  # ns-pointer str-msg-type
    loglevel: str = 'cancel',
-    **ep_kwargs,
+    **kwargs,
 ) -> None:
-
+    get_console_log('info', name=__name__)
    log = get_console_log(
        level=loglevel,
        name='piker.service',
    )
    async with open_docker() as client:
@ -358,117 +300,38 @@ async def open_ahabd(
        (
            dcntr,
            cntr_config,
-            start_pred,
+            start_msg,
-            stop_pred,
+            stop_msg,
-        ) = ep_func(client, **ep_kwargs)
+        ) = ep_func(client)
        cntr = Container(dcntr)
-        conf: ChainMap[str, Any] = ChainMap(
+        with trio.move_on_after(1):
            found = await cntr.process_logs_until(start_msg)
-            # container specific
+            if not found and cntr not in client.containers.list():
                raise RuntimeError(
                    'Failed to start `marketstore` check logs deats'
                )
        await ctx.started((
            cntr.cntr.id,
            os.getpid(),
            cntr_config,
-
+        ))
            # defaults
            {
                # startup time limit which is the max the supervisor
                # will wait for the container to be registered in
                # ``client.containers.list()``
                'startup_timeout': 1.0,
                # how fast to poll for the starup predicate by sleeping
                # this amount incrementally thus yielding to the
                # ``trio`` scheduler on during sync polling execution.
                'startup_query_period': 0.001,
                # str-key value expected to contain log message body-contents
                # when read using:
                # ``json.loads(entry for entry in DockerContainer.logs())``
                'log_msg_key': 'msg',
                # startup sync func, like `Nursery.started()`
                'started_afunc': None,
            },
        )
        try:
            with trio.move_on_after(conf['startup_timeout']) as cs:
                async with trio.open_nursery() as tn:
                    tn.start_soon(
                        partial(
                            cntr.process_logs_until,
                            log_msg_key=conf['log_msg_key'],
                            patt_matcher=start_pred,
                            checkpoint_period=conf['startup_query_period'],
                        )
                    )
                    # optional blocking routine
                    started = conf['started_afunc']
                    if started:
                        await started()
                    # poll for container startup or timeout
                    while not cs.cancel_called:
                        if dcntr in client.containers.list():
                            break
                        await trio.sleep(conf['startup_query_period'])
                    # sync with remote caller actor-task but allow log
                    # processing to continue running in bg.
                    await ctx.started((
                        cntr.cntr.id,
                        os.getpid(),
                        cntr_config,
                    ))
                # XXX: if we timeout on finding the "startup msg" we
                # expect then we want to FOR SURE raise an error
                # upwards!
                if cs.cancelled_caught:
                    # if dcntr not in client.containers.list():
                    for entry in cntr.seen_so_far:
                        log.info(entry)
                    raise DockerNotStarted(
                        f'Failed to start container: {cntr.cuid}\n'
                        f'due to timeout={conf["startup_timeout"]}s\n\n'
                        "check ur container's logs!"
                    )
            # TODO: we might eventually want a proxy-style msg-prot here
            # to allow remote control of containers without needing
            # callers to have root perms?
            await trio.sleep_forever()
        finally:
-            # TODO: ensure loglevel can be set and teardown logs are
+            await cntr.cancel(stop_msg)
            # reported if possible on error or cancel..
            # XXX WARNING: currently shielding here can result in hangs
            # on ctl-c from user.. ideally we can avoid a cancel getting
            # consumed and not propagating whilst still doing teardown
            # logging..
            with trio.CancelScope(shield=True):
                await cntr.cancel(
                    log_msg_key=conf['log_msg_key'],
                    stop_predicate=stop_pred,
                )
-@acm
+async def start_ahab(
 async def start_ahab_service(
    services: Services,
    service_name: str,
    # endpoint config passed as **kwargs
    endpoint: Callable[docker.DockerClient, DockerContainer],
    ep_kwargs: dict,
    loglevel: str | None = 'cancel',
    # supervisor config
    drop_root_perms: bool = True,
    task_status: TaskStatus[
        tuple[
            trio.Event,
@ -487,17 +350,15 @@ async def start_ahab_service(
    is started.
    '''
    # global log
    log = get_console_log(loglevel or 'cancel')
    cn_ready = trio.Event()
    try:
-        async with tractor.open_nursery() as an:
+        async with tractor.open_nursery(
            loglevel='runtime',
        ) as tn:
-            portal = await an.start_actor(
+            portal = await tn.start_actor(
                service_name,
-                enable_modules=[__name__],
+                enable_modules=[__name__]
                loglevel=loglevel,
            )
            # TODO: we have issues with this on teardown
@ -507,10 +368,7 @@ async def start_ahab_service(
            # de-escalate root perms to the original user
            # after the docker supervisor actor is spawned.
-            if (
+            if config._parent_user:
                drop_root_perms
                and config._parent_user
            ):
                import pwd
                os.setuid(
                    pwd.getpwnam(
@ -518,28 +376,20 @@ async def start_ahab_service(
                    )[2]  # named user's uid
                )
-            cs, first = await services.start_service_task(
+            async with portal.open_context(
-                name=service_name,
+                open_ahabd,
                portal=portal,
                # rest: endpoint inputs
                target=open_ahabd,
                endpoint=str(NamespacePath.from_ref(endpoint)),
-                loglevel='cancel',
+            ) as (ctx, first):
                **ep_kwargs,
            )
-            cid, pid, cntr_config = first
+                cid, pid, cntr_config = first
-            try:
+                task_status.started((
                yield (
                    cn_ready,
                    cntr_config,
                    (cid, pid),
-                )
+                ))
-            finally:
+
-                log.info(f'Cancelling ahab service `{service_name}`')
+                await trio.sleep_forever()
                await services.cancel_service(service_name)
    # since we demoted root perms in this parent
    # we'll get a perms error on proc cleanup in
--- a/piker/data/_normalize.py
+++ b/piker/data/_normalize.py
@ -0,0 +1,82 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Stream format enforcement.
 '''
 from itertools import chain
 from typing import AsyncIterator
 def iterticks(
    quote: dict,
    types: tuple[str] = (
        'trade',
        'dark_trade',
    ),
    deduplicate_darks: bool = False,
 ) -> AsyncIterator:
    '''
    Iterate through ticks delivered per quote cycle.
    '''
    if deduplicate_darks:
        assert 'dark_trade' in types
    # print(f"{quote}\n\n")
    ticks = quote.get('ticks', ())
    trades = {}
    darks = {}
    if ticks:
        # do a first pass and attempt to remove duplicate dark
        # trades with the same tick signature.
        if deduplicate_darks:
            for tick in ticks:
                ttype = tick.get('type')
                time = tick.get('time', None)
                if time:
                    sig = (
                        time,
                        tick['price'],
                        tick.get('size')
                    )
                    if ttype == 'dark_trade':
                        darks[sig] = tick
                    elif ttype == 'trade':
                        trades[sig] = tick
            # filter duplicates
            for sig, tick in trades.items():
                tick = darks.pop(sig, None)
                if tick:
                    ticks.remove(tick)
                    # print(f'DUPLICATE {tick}')
            # re-insert ticks
            ticks.extend(list(chain(trades.values(), darks.values())))
        for tick in ticks:
            # print(f"{quote['symbol']}: {tick}")
            ttype = tick.get('type')
            if ttype in types:
                yield tick
--- a/piker/data/_pathops.py
+++ b/piker/data/_pathops.py
@ -1,281 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) 2018-present  Tyler Goodlet (in stewardship of pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 Super fast ``QPainterPath`` generation related operator routines.
 """
 import numpy as np
 from numpy.lib import recfunctions as rfn
 from numba import (
    # types,
    njit,
    float64,
    int64,
    # optional,
 )
 # TODO: for ``numba`` typing..
 # from ._source import numba_ohlc_dtype
 from ._m4 import ds_m4
 def xy_downsample(
    x,
    y,
    uppx,
    x_spacer: float = 0.5,
 ) -> tuple[
    np.ndarray,
    np.ndarray,
    float,
    float,
 ]:
    '''
    Downsample 1D (flat ``numpy.ndarray``) arrays using M4 given an input
    ``uppx`` (units-per-pixel) and add space between discreet datums.
    '''
    # downsample whenever more then 1 pixels per datum can be shown.
    # always refresh data bounds until we get diffing
    # working properly, see above..
    m4_out = ds_m4(
        x,
        y,
        uppx,
    )
    if m4_out is not None:
        bins, x, y, ymn, ymx = m4_out
        # flatten output to 1d arrays suitable for path-graphics generation.
        x = np.broadcast_to(x[:, None], y.shape)
        x = (x + np.array(
            [-x_spacer, 0, 0, x_spacer]
        )).flatten()
        y = y.flatten()
        return x, y, ymn, ymx
    # XXX: we accept a None output for the case where the input range
    # to ``ds_m4()`` is bad (-ve) and we want to catch and debug
    # that (seemingly super rare) circumstance..
    return None
@njit(
    # NOTE: need to construct this manually for readonly
    # arrays, see https://github.com/numba/numba/issues/4511
    # (
    #     types.Array(
    #         numba_ohlc_dtype,
    #         1,
    #         'C',
    #         readonly=True,
    #     ),
    #     int64,
    #     types.unicode_type,
    #     optional(float64),
    # ),
    nogil=True
 )
 def path_arrays_from_ohlc(
    data: np.ndarray,
    start: int64,
    bar_w: float64,
    bar_gap: float64 = 0.16,
    use_time_index: bool = True,
    # XXX: ``numba`` issue: https://github.com/numba/numba/issues/8622
    # index_field: str,
 ) -> tuple[
    np.ndarray,
    np.ndarray,
    np.ndarray,
 ]:
    '''
    Generate an array of lines objects from input ohlc data.
    '''
    size = int(data.shape[0] * 6)
    # XXX: see this for why the dtype might have to be defined outside
    # the routine.
    # https://github.com/numba/numba/issues/4098#issuecomment-493914533
    x = np.zeros(
        shape=size,
        dtype=float64,
    )
    y, c = x.copy(), x.copy()
    half_w: float = bar_w/2
    # TODO: report bug for assert @
    # ../piker/env/lib/python3.8/site-packages/numba/core/typing/builtins.py:991
    for i, q in enumerate(data[start:], start):
        open = q['open']
        high = q['high']
        low = q['low']
        close = q['close']
        if use_time_index:
            index = float64(q['time'])
        else:
            index = float64(q['index'])
        # XXX: ``numba`` issue: https://github.com/numba/numba/issues/8622
        # index = float64(q[index_field])
        # AND this (probably)
        # open, high, low, close, index = q[
        #     ['open', 'high', 'low', 'close', 'index']]
        istart = i * 6
        istop = istart + 6
        # x,y detail the 6 points which connect all vertexes of a ohlc bar
        mid: float = index + half_w
        x[istart:istop] = (
            index + bar_gap,
            mid,
            mid,
            mid,
            mid,
            index + bar_w - bar_gap,
        )
        y[istart:istop] = (
            open,
            open,
            low,
            high,
            close,
            close,
        )
        # specifies that the first edge is never connected to the
        # prior bars last edge thus providing a small "gap"/"space"
        # between bars determined by ``bar_gap``.
        c[istart:istop] = (1, 1, 1, 1, 1, 0)
    return x, y, c
 def hl2mxmn(
    ohlc: np.ndarray,
    index_field: str = 'index',
 ) -> np.ndarray:
    '''
    Convert a OHLC struct-array containing 'high'/'low' columns
    to a "joined" max/min 1-d array.
    '''
    index = ohlc[index_field]
    hls = ohlc[[
        'low',
        'high',
    ]]
    mxmn = np.empty(2*hls.size, dtype=np.float64)
    x = np.empty(2*hls.size, dtype=np.float64)
    trace_hl(hls, mxmn, x, index[0])
    x = x + index[0]
    return mxmn, x
@njit(
    # TODO: the type annots..
    # float64[:](float64[:],),
 )
 def trace_hl(
    hl: 'np.ndarray',
    out: np.ndarray,
    x: np.ndarray,
    start: int,
    # the "offset" values in the x-domain which
    # place the 2 output points around each ``int``
    # master index.
    margin: float = 0.43,
 ) -> None:
    '''
    "Trace" the outline of the high-low values of an ohlc sequence
    as a line such that the maximum deviation (aka disperaion) between
    bars if preserved.
    This routine is expected to modify input arrays in-place.
    '''
    last_l = hl['low'][0]
    last_h = hl['high'][0]
    for i in range(hl.size):
        row = hl[i]
        lo, hi = row['low'], row['high']
        up_diff = hi - last_l
        down_diff = last_h - lo
        if up_diff > down_diff:
            out[2*i + 1] = hi
            out[2*i] = last_l
        else:
            out[2*i + 1] = lo
            out[2*i] = last_h
        last_l = lo
        last_h = hi
        x[2*i] = int(i) - margin
        x[2*i + 1] = int(i) + margin
    return out
 def ohlc_flatten(
    ohlc: np.ndarray,
    use_mxmn: bool = True,
    index_field: str = 'index',
 ) -> tuple[np.ndarray, np.ndarray]:
    '''
    Convert an OHLCV struct-array into a flat ready-for-line-plotting
    1-d array that is 4 times the size with x-domain values distributed
    evenly (by 0.5 steps) over each index.
    '''
    index = ohlc[index_field]
    if use_mxmn:
        # traces a line optimally over highs to lows
        # using numba. NOTE: pretty sure this is faster
        # and looks about the same as the below output.
        flat, x = hl2mxmn(ohlc)
    else:
        flat = rfn.structured_to_unstructured(
            ohlc[['open', 'high', 'low', 'close']]
        ).flatten()
        x = np.linspace(
            start=index[0] - 0.5,
            stop=index[-1] + 0.5,
            num=len(flat),
        )
    return x, flat
--- a/piker/data/_sampling.py
+++ b/piker/data/_sampling.py
@ -27,41 +27,30 @@ from collections import (
 from contextlib import asynccontextmanager as acm
 import time
 from typing import (
    Any,
    AsyncIterator,
    TYPE_CHECKING,
 )
 import tractor
 from tractor import (
    Context,
    MsgStream,
    Channel,
 )
 from tractor.trionics import (
    maybe_open_nursery,
 )
 import trio
 from trio_typing import TaskStatus
-from .ticktools import (
+from ..log import (
-    frame_ticks,
+    get_logger,
    _tick_groups,
 )
 from ._util import (
    log,
    get_console_log,
 )
-from ..service import maybe_spawn_daemon
+from .._daemon import maybe_spawn_daemon
 if TYPE_CHECKING:
    from ._sharedmem import (
        ShmArray,
    )
-    from .feed import (
+    from .feed import _FeedsBus
-        _FeedsBus,
+
-        Sub,
+log = get_logger(__name__)
    )
 # highest frequency sample step is 1 second by default, though in
@ -79,37 +68,31 @@ class Sampler:
    This non-instantiated type is meant to be a singleton within
    a `samplerd` actor-service spawned once by the user wishing to
-    time-step-sample (real-time) quote feeds, see
+    time-step sample real-time quote feeds, see
-    `.service.maybe_open_samplerd()` and the below
+    ``._daemon.maybe_open_samplerd()`` and the below
-    `register_with_sampler()`.
+    ``register_with_sampler()``.
    '''
-    service_nursery: None|trio.Nursery = None
+    service_nursery: None | trio.Nursery = None
-    # TODO: we could stick these in a composed type to avoid angering
+    # TODO: we could stick these in a composed type to avoid
-    # the "i hate module scoped variables crowd" (yawn).
+    # angering the "i hate module scoped variables crowd" (yawn).
    ohlcv_shms: dict[float, list[ShmArray]] = {}
    # holds one-task-per-sample-period tasks which are spawned as-needed by
    # data feed requests with a given detected time step usually from
    # history loading.
-    incr_task_cs: trio.CancelScope|None = None
+    incr_task_cs: trio.CancelScope | None = None
    bcast_errors: tuple[Exception] = (
        trio.BrokenResourceError,
        trio.ClosedResourceError,
        trio.EndOfChannel,
        tractor.TransportClosed,
    )
    # holds all the ``tractor.Context`` remote subscriptions for
    # a particular sample period increment event: all subscribers are
    # notified on a step.
    # subscribers: dict[int, list[tractor.MsgStream]] = {}
    subscribers: defaultdict[
        float,
        list[
            float,
-            set[MsgStream]
+            set[tractor.MsgStream]
        ],
    ] = defaultdict(
        lambda: [
@ -249,8 +232,7 @@ class Sampler:
    async def broadcast(
        self,
        period_s: float,
-        time_stamp: float|None = None,
+        time_stamp: float | None = None,
        info: dict|None = None,
    ) -> None:
        '''
@ -258,96 +240,60 @@ class Sampler:
        subscribers for a given sample period.
        '''
        pair: list[float, set]
        pair = self.subscribers[period_s]
        last_ts: float
        subs: set
        last_ts, subs = pair
-        # NOTE, for debugging pub-sub issues
+        task = trio.lowlevel.current_task()
-        # task = trio.lowlevel.current_task()
+        log.debug(
-        # log.debug(
+            f'SUBS {self.subscribers}\n'
-        #     f'AlL-SUBS@{period_s!r}: {self.subscribers}\n'
+            f'PAIR {pair}\n'
-        #     f'PAIR: {pair}\n'
+            f'TASK: {task}: {id(task)}\n'
-        #     f'TASK: {task}: {id(task)}\n'
+            f'broadcasting {period_s} -> {last_ts}\n'
-        #     f'broadcasting {period_s} -> {last_ts}\n'
+            # f'consumers: {subs}'
-        #     f'consumers: {subs}'
+        )
-        # )
+        borked: set[tractor.MsgStream] = set()
-        borked: set[MsgStream] = set()
+        for stream in subs:
        sent: set[MsgStream] = set()
        while True:
            try:
-                for stream in (subs - sent):
+                await stream.send({
-                    try:
+                    'index': time_stamp or last_ts,
-                        msg = {
+                    'period': period_s,
-                            'index': time_stamp or last_ts,
+                })
-                            'period': period_s,
+            except (
-                        }
+                trio.BrokenResourceError,
-                        if info:
+                trio.ClosedResourceError
-                            msg.update(info)
+            ):
-
+                log.error(
-                        await stream.send(msg)
+                    f'{stream._ctx.chan.uid} dropped connection'
-                        sent.add(stream)
+                )
-
+                borked.add(stream)
                    except self.bcast_errors as err:
                        log.error(
                            f'Connection dropped for IPC ctx due to,\n'
                            f'{type(err)!r}\n'
                            f'\n'
                            f'{stream._ctx}'
                        )
                        borked.add(stream)
                else:
                    break
            except RuntimeError:
                log.warning(f'Client subs {subs} changed while broadcasting')
                continue
        for stream in borked:
            try:
                subs.remove(stream)
-            except KeyError:
+            except ValueError:
                log.warning(
                    f'{stream._ctx.chan.uid} sub already removed!?'
                )
    @classmethod
-    async def broadcast_all(
+    async def broadcast_all(self) -> None:
-        self,
+        for period_s in self.subscribers:
-        info: dict|None = None,
+            await self.broadcast(period_s)
    ) -> None:
        # NOTE: take a copy of subs since removals can happen
        # during the broadcast checkpoint which can cause
        # a `RuntimeError` on interation of the underlying `dict`.
        for period_s in list(self.subscribers):
            await self.broadcast(
                period_s,
                info=info,
            )
@tractor.context
 async def register_with_sampler(
-    ctx: Context,
+    ctx: tractor.Context,
    period_s: float,
-    shms_by_period: dict[float, dict]|None = None,
+    shms_by_period: dict[float, dict] | None = None,
    open_index_stream: bool = True,  # open a 2way stream for sample step msgs?
    sub_for_broadcasts: bool = True,  # sampler side to send step updates?
    loglevel: str|None = None,
-) -> set[int]:
+) -> None:
-    get_console_log(
+    get_console_log(tractor.current_actor().loglevel)
        level=(
            loglevel
            or
            tractor.current_actor().loglevel
        ),
        name=__name__,
    )
    incr_was_started: bool = False
    try:
@ -372,12 +318,7 @@ async def register_with_sampler(
            # insert the base 1s period (for OHLC style sampling) into
            # the increment buffer set to update and shift every second.
-            if (
+            if shms_by_period is not None:
                shms_by_period is not None
                # and
                # feed_is_live.is_set()
                # ^TODO? pass it in instead?
            ):
                from ._sharedmem import (
                    attach_shm_array,
                    _Token,
@ -391,44 +332,26 @@ async def register_with_sampler(
                        readonly=False,
                    )
                    shms_by_period[period] = shm
-                    Sampler.ohlcv_shms.setdefault(
+                    Sampler.ohlcv_shms.setdefault(period, []).append(shm)
                        period,
                        [],
                    ).append(shm)
                assert Sampler.ohlcv_shms
            # unblock caller
-            await ctx.started(
+            await ctx.started(set(Sampler.ohlcv_shms.keys()))
                set(Sampler.ohlcv_shms.keys())
            )
            if open_index_stream:
                try:
-                    async with ctx.open_stream(
+                    async with ctx.open_stream() as stream:
                        allow_overruns=True,
                    ) as stream:
                        if sub_for_broadcasts:
                            subs.add(stream)
                        # except broadcast requests from the subscriber
                        async for msg in stream:
-                            if 'broadcast_all' in msg:
+                            if msg == 'broadcast_all':
-                                await Sampler.broadcast_all(
+                                await Sampler.broadcast_all()
                                    info=msg['broadcast_all'],
                                )
                finally:
-                    if (
+                    if sub_for_broadcasts:
-                        sub_for_broadcasts
+                        subs.remove(stream)
                        and
                        subs
                    ):
                        try:
                            subs.remove(stream)
                        except KeyError:
                            log.warning(
                                f'{stream._ctx.chan.uid} sub already removed!?'
                            )
            else:
                # if no shms are passed in we just wait until cancelled
                # by caller.
@ -447,7 +370,7 @@ async def register_with_sampler(
 async def spawn_samplerd(
-    loglevel: str|None = None,
+    loglevel: str | None = None,
    **extra_tractor_kwargs
 ) -> bool:
@ -456,7 +379,7 @@ async def spawn_samplerd(
    update and increment count write and stream broadcasting.
    '''
-    from piker.service import Services
+    from piker._daemon import Services
    dname = 'samplerd'
    log.info(f'Spawning `{dname}`')
@ -484,7 +407,6 @@ async def spawn_samplerd(
                register_with_sampler,
                period_s=1,
                sub_for_broadcasts=False,
                loglevel=loglevel,
            )
            return True
@ -493,10 +415,11 @@ async def spawn_samplerd(
@acm
 async def maybe_open_samplerd(
    loglevel: str|None = None,
    **pikerd_kwargs,
-) -> tractor.Portal:  # noqa
+    loglevel: str | None = None,
    **kwargs,
 ) -> tractor._portal.Portal:  # noqa
    '''
    Client-side helper to maybe startup the ``samplerd`` service
    under the ``pikerd`` tree.
@ -507,9 +430,9 @@ async def maybe_open_samplerd(
    async with maybe_spawn_daemon(
        dname,
        service_task_target=spawn_samplerd,
-        spawn_args={},
+        spawn_args={'loglevel': loglevel},
        loglevel=loglevel,
-        **pikerd_kwargs,
+        **kwargs,
    ) as portal:
        yield portal
@ -518,14 +441,12 @@ async def maybe_open_samplerd(
@acm
 async def open_sample_stream(
    period_s: float,
-    shms_by_period: dict[float, dict]|None = None,
+    shms_by_period: dict[float, dict] | None = None,
    open_index_stream: bool = True,
    sub_for_broadcasts: bool = True,
    loglevel: str|None = None,
-    # cache_key: str|None = None,
+    cache_key: str | None = None,
-    # allow_new_sampler: bool = True,
+    allow_new_sampler: bool = True,
    ensure_is_active: bool = False,
 ) -> AsyncIterator[dict[str, float]]:
    '''
@ -553,15 +474,11 @@ async def open_sample_stream(
    #         yield bistream
    # else:
    ctx: tractor.Context
    shm_periods: set[int]  # in `int`-seconds
    async with (
        # XXX: this should be singleton on a host,
        # a lone broker-daemon per provider should be
        # created for all practical purposes
-        maybe_open_samplerd(
+        maybe_open_samplerd() as portal,
            loglevel=loglevel,
        ) as portal,
        portal.open_context(
            register_with_sampler,
@ -570,30 +487,21 @@ async def open_sample_stream(
                'shms_by_period': shms_by_period,
                'open_index_stream': open_index_stream,
                'sub_for_broadcasts': sub_for_broadcasts,
                'loglevel': loglevel,
            },
-        ) as (ctx, shm_periods)
+        ) as (ctx, first)
    ):
        if ensure_is_active:
            assert len(shm_periods) > 1
        async with (
-            ctx.open_stream(
+            ctx.open_stream() as istream,
                allow_overruns=True,
            ) as istream,
-            # TODO: we DO need this task-bcasting so that
+            # TODO: we don't need this task-bcasting right?
-            # for eg. the history chart update loop eventually
+            # istream.subscribe() as istream,
            # receceives all backfilling event msgs such that
            # the underlying graphics format arrays are
            # re-allocated until all history is loaded!
            istream.subscribe() as istream,
        ):
            yield istream
 async def sample_and_broadcast(
-    bus: _FeedsBus,
+
    bus: _FeedsBus,  # noqa
    rt_shm: ShmArray,
    hist_shm: ShmArray,
    quote_stream: trio.abc.ReceiveChannel,
@ -613,33 +521,11 @@ async def sample_and_broadcast(
    overruns = Counter()
    # NOTE, only used for debugging live-data-feed issues, though
    # this should be resolved more correctly in the future using the
    # new typed-msgspec feats of `tractor`!
    #
    # XXX, a multiline nested `dict` formatter (since rn quote-msgs
    # are just that).
    # pfmt: Callable[[str], str] = mk_repr()
    # iterate stream delivered by broker
    async for quotes in quote_stream:
        # print(quotes)
-        # XXX WARNING XXX only enable for debugging bc ow can cost
+        # TODO: ``numba`` this!
        # ALOT of perf with HF-feedz!!!
        #
        # log.info(
        #     'Rx live quotes:\n'
        #     f'{pfmt(quotes)}'
        # )
        # TODO,
        # -[ ] `numba` or `cython`-nize this loop possibly?
        #  |_alternatively could we do it in rust somehow by upacking
        #    arrow msgs instead of using `msgspec`?
        # -[ ] use `msgspec.Struct` support in new typed-msging from
        #     `tractor` to ensure only allowed msgs are transmitted?
        #
        for broker_symbol, quote in quotes.items():
            # TODO: in theory you can send the IPC msg *before* writing
            # to the sharedmem array to decrease latency, however, that
@ -653,9 +539,9 @@ async def sample_and_broadcast(
            # TODO: we should probably not write every single
            # value to an OHLC sample stream XD
            # for a tick stream sure.. but this is excessive..
-            ticks: list[dict] = quote['ticks']
+            ticks = quote['ticks']
            for tick in ticks:
-                ticktype: str = tick['type']
+                ticktype = tick['type']
                # write trade events to shm last OHLC sample
                if ticktype in ('trade', 'utrade'):
@ -665,14 +551,13 @@ async def sample_and_broadcast(
                    # more compact inline-way to do this assignment
                    # to both buffers?
                    for shm in [rt_shm, hist_shm]:
                        # update last entry
                        # benchmarked in the 4-5 us range
                        o, high, low, v = shm.array[-1][
                            ['open', 'high', 'low', 'volume']
                        ]
-                        new_v: float = tick.get('size', 0)
+                        new_v = tick.get('size', 0)
                        if v == 0 and new_v:
                            # no trades for this bar yet so the open
@ -691,14 +576,14 @@ async def sample_and_broadcast(
                            'high',
                            'low',
                            'close',
-                            # 'bar_wap',  # can be optionally provided
+                            'bar_wap',  # can be optionally provided
                            'volume',
                        ]][-1] = (
                            o,
                            max(high, last),
                            min(low, last),
                            last,
-                            # quote.get('bar_wap', 0),
+                            quote.get('bar_wap', 0),
                            volume,
                        )
@ -710,64 +595,40 @@ async def sample_and_broadcast(
            # eventually block this producer end of the feed and
            # thus other consumers still attached.
            sub_key: str = broker_symbol.lower()
-            subs: set[Sub] = bus.get_subs(sub_key)
+            subs: list[
-
+                tuple[
-            # TODO, figure out how to make this useful whilst
+                    tractor.MsgStream | trio.MemorySendChannel,
-            # incoporating feed "pausing" ..
+                    float | None,  # tick throttle in Hz
-            #
+                ]
-            # if not subs:
+            ] = bus.get_subs(sub_key)
            #     all_bs_fqmes: list[str] = list(
            #         bus._subscribers.keys()
            #     )
            #     log.warning(
            #         f'No subscribers for {brokername!r} live-quote ??\n'
            #         f'broker_symbol: {broker_symbol}\n\n'
            #         f'Maybe the backend-sys symbol does not match one of,\n'
            #         f'{pfmt(all_bs_fqmes)}\n'
            #     )
            # NOTE: by default the broker backend doesn't append
-            # it's own "name" into the fqme schema (but maybe it
+            # it's own "name" into the fqsn schema (but maybe it
            # should?) so we have to manually generate the correct
            # key here.
-            fqme: str = f'{broker_symbol}.{brokername}'
+            fqsn = f'{broker_symbol}.{brokername}'
            lags: int = 0
-            # XXX TODO XXX: speed up this loop in an AOT compiled
+            for (stream, tick_throttle) in subs.copy():
            # lang (like rust or nim or zig)!
            # AND/OR instead of doing a fan out to TCP sockets
            # here, we add a shm-style tick queue which readers can
            # pull from instead of placing the burden of broadcast
            # on solely on this `brokerd` actor. see issues:
            # - https://github.com/pikers/piker/issues/98
            # - https://github.com/pikers/piker/issues/107
            # for (stream, tick_throttle) in subs.copy():
            for sub in subs.copy():
                ipc: MsgStream = sub.ipc
                throttle: float = sub.throttle_rate
                try:
                    with trio.move_on_after(0.2) as cs:
-                        if throttle:
+                        if tick_throttle:
                            send_chan: trio.abc.SendChannel = sub.send_chan
                            # this is a send mem chan that likely
                            # pushes to the ``uniform_rate_send()`` below.
                            try:
-                                send_chan.send_nowait(
+                                stream.send_nowait(
-                                    (fqme, quote)
+                                    (fqsn, quote)
                                )
                            except trio.WouldBlock:
                                overruns[sub_key] += 1
-                                ctx: Context = ipc._ctx
+                                ctx = stream._ctx
-                                chan: Channel = ctx.chan
+                                chan = ctx.chan
                                log.warning(
                                    f'Feed OVERRUN {sub_key}'
-                                    f'@{bus.brokername} -> \n'
+                                    '@{bus.brokername} -> \n'
-                                    f'feed @ {chan.aid.reprol()}\n'
+                                    f'feed @ {chan.uid}\n'
-                                    f'throttle = {throttle} Hz'
+                                    f'throttle = {tick_throttle} Hz'
                                )
                                if overruns[sub_key] > 6:
@ -784,29 +645,33 @@ async def sample_and_broadcast(
                                            f'{sub_key}:'
                                            f'{ctx.cid}@{chan.uid}'
                                        )
-                                        await ipc.aclose()
+                                        await stream.aclose()
                                        raise trio.BrokenResourceError
                        else:
-                            await ipc.send(
+                            await stream.send(
-                                {fqme: quote}
+                                {fqsn: quote}
                            )
                    if cs.cancelled_caught:
                        lags += 1
                        if lags > 10:
-                            await tractor.pause()
+                            await tractor.breakpoint()
-                except Sampler.bcast_errors as ipc_err:
+                except (
-                    ctx: Context = ipc._ctx
+                    trio.BrokenResourceError,
-                    chan: Channel = ctx.chan
+                    trio.ClosedResourceError,
                    trio.EndOfChannel,
                ):
                    ctx = stream._ctx
                    chan = ctx.chan
                    if ctx:
                        log.warning(
-                            f'Dropped `brokerd`-feed for {broker_symbol!r} due to,\n'
+                            'Dropped `brokerd`-quotes-feed connection:\n'
-                            f'x>) {ctx.cid}@{chan.uid}'
+                            f'{broker_symbol}:'
-                            f'|_{ipc_err!r}\n\n'
+                            f'{ctx.cid}@{chan.uid}'
                        )
-                    if sub.throttle_rate:
+                    if tick_throttle:
-                        assert ipc._closed
+                        assert stream._closed
                    # XXX: do we need to deregister here
                    # if it's done in the fee bus code?
@ -815,75 +680,113 @@ async def sample_and_broadcast(
                    # since there seems to be some kinda race..
                    bus.remove_subs(
                        sub_key,
-                        {sub},
+                        {(stream, tick_throttle)},
                    )
 # a working tick-type-classes template
 _tick_groups = {
    'clears': {'trade', 'dark_trade', 'last'},
    'bids': {'bid', 'bsize'},
    'asks': {'ask', 'asize'},
 }
 def frame_ticks(
    first_quote: dict,
    last_quote: dict,
    ticks_by_type: dict,
 ) -> None:
    # append quotes since last iteration into the last quote's
    # tick array/buffer.
    ticks = last_quote.get('ticks')
    # TODO: once we decide to get fancy really we should
    # have a shared mem tick buffer that is just
    # continually filled and the UI just ready from it
    # at it's display rate.
    if ticks:
        # TODO: do we need this any more or can we just
        # expect the receiver to unwind the below
        # `ticks_by_type: dict`?
        # => undwinding would potentially require a
        # `dict[str, set | list]` instead with an
        # included `'types' field which is an (ordered)
        # set of tick type fields in the order which
        # types arrived?
        first_quote['ticks'].extend(ticks)
        # XXX: build a tick-by-type table of lists
        # of tick messages. This allows for less
        # iteration on the receiver side by allowing for
        # a single "latest tick event" look up by
        # indexing the last entry in each sub-list.
        # tbt = {
        #     'types': ['bid', 'asize', 'last', .. '<type_n>'],
        #     'bid': [tick0, tick1, tick2, .., tickn],
        #     'asize': [tick0, tick1, tick2, .., tickn],
        #     'last': [tick0, tick1, tick2, .., tickn],
        #     ...
        #     '<type_n>': [tick0, tick1, tick2, .., tickn],
        # }
        # append in reverse FIFO order for in-order iteration on
        # receiver side.
        for tick in ticks:
            ttype = tick['type']
            ticks_by_type[ttype].append(tick)
 # TODO: a less naive throttler, here's some snippets:
 # token bucket by njs:
 # https://gist.github.com/njsmith/7ea44ec07e901cb78ebe1dd8dd846cb9
 async def uniform_rate_send(
    rate: float,
    quote_stream: trio.abc.ReceiveChannel,
-    stream: MsgStream,
+    stream: tractor.MsgStream,
-    task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
+    task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
 ) -> None:
    '''
    Throttle a real-time (presumably tick event) stream to a uniform
    transmissiom rate, normally for the purposes of throttling a data
    flow being consumed by a graphics rendering actor which itself is limited
    by a fixed maximum display rate.
-    Though this function isn't documented (nor was intentially written
+    # try not to error-out on overruns of the subscribed (chart) client
-    to be) a token-bucket style algo, it effectively operates as one (we
+    stream._ctx._backpressure = True
    think?).
-    TODO: a less naive throttler, here's some snippets:
+    # TODO: compute the approx overhead latency per cycle
-    token bucket by njs:
+    left_to_sleep = throttle_period = 1/rate - 0.000616
    https://gist.github.com/njsmith/7ea44ec07e901cb78ebe1dd8dd846cb9
    '''
    # ?TODO? dynamically compute the **actual** approx overhead latency per cycle
    # instead of this magic # bidinezz?
    throttle_period: float = 1/rate - 0.000616
    left_to_sleep: float = throttle_period
    # send cycle state
    first_quote: dict|None
    first_quote = last_quote = None
-    last_send: float = time.time()
+    last_send = time.time()
-    diff: float = 0
+    diff = 0
    task_status.started()
-    ticks_by_type: dict[
+    ticks_by_type: defaultdict[
        str,
-        list[dict[str, Any]],
+        list[dict],
-    ] = {}
+    ] = defaultdict(list)
    clear_types = _tick_groups['clears']
    while True:
        # compute the remaining time to sleep for this throttled cycle
-        left_to_sleep: float = throttle_period - diff
+        left_to_sleep = throttle_period - diff
        if left_to_sleep > 0:
            cs: trio.CancelScope
            with trio.move_on_after(left_to_sleep) as cs:
                sym: str
                last_quote: dict
                try:
                    sym, last_quote = await quote_stream.receive()
                except trio.EndOfChannel:
-                    log.exception(
+                    log.exception(f"feed for {stream} ended?")
                        f'Live stream for feed for ended?\n'
                        f'<=c\n'
                        f'  |_[{stream!r}\n'
                    )
                    break
-                diff: float = time.time() - last_send
+                diff = time.time() - last_send
                if not first_quote:
-                    first_quote: float = last_quote
+                    first_quote = last_quote
                    # first_quote['tbt'] = ticks_by_type
                if (throttle_period - diff) > 0:
@ -891,9 +794,9 @@ async def uniform_rate_send(
                    # expired we aren't supposed to send yet so append
                    # to the tick frame.
                    frame_ticks(
                        first_quote,
                        last_quote,
-                        ticks_in_order=first_quote['ticks'],
+                        ticks_by_type,
                        ticks_by_type=ticks_by_type,
                    )
                    # send cycle isn't due yet so continue waiting
@ -913,8 +816,8 @@ async def uniform_rate_send(
                frame_ticks(
                    first_quote,
-                    ticks_in_order=first_quote['ticks'],
+                    first_quote,
-                    ticks_by_type=ticks_by_type,
+                    ticks_by_type,
                )
            # we have a quote already so send it now.
@ -930,9 +833,9 @@ async def uniform_rate_send(
                    break
                frame_ticks(
                    first_quote,
                    last_quote,
-                    ticks_in_order=first_quote['ticks'],
+                    ticks_by_type,
                    ticks_by_type=ticks_by_type,
                )
        # measured_rate = 1 / (time.time() - last_send)
@ -944,41 +847,20 @@ async def uniform_rate_send(
        # TODO: now if only we could sync this to the display
        # rate timing exactly lul
        try:
-            await stream.send({
+            await stream.send({sym: first_quote})
                sym: first_quote
            })
        except tractor.RemoteActorError as rme:
            if rme.type is not tractor._exceptions.StreamOverrun:
                raise
            ctx = stream._ctx
            chan = ctx.chan
            log.warning(
                'Throttled quote-stream overrun!\n'
                f'{sym}:{ctx.cid}@{chan.uid}'
            )
        # NOTE: any of these can be raised by `tractor`'s IPC
        # transport-layer and we want to be highly resilient
        # to consumers which crash or lose network connection.
        # I.e. we **DO NOT** want to crash and propagate up to
        # ``pikerd`` these kinds of errors!
        except (
            # NOTE: any of these can be raised by ``tractor``'s IPC
            # transport-layer and we want to be highly resilient
            # to consumers which crash or lose network connection.
            # I.e. we **DO NOT** want to crash and propagate up to
            # ``pikerd`` these kinds of errors!
            trio.ClosedResourceError,
            trio.BrokenResourceError,
            ConnectionResetError,
-        ) + Sampler.bcast_errors as ipc_err:
+        ):
-            match ipc_err:
+            # if the feed consumer goes down then drop
-                case trio.EndOfChannel():
+            # out of this rate limiter
-                    log.info(
+            log.warning(f'{stream} closed')
                        f'{stream} terminated by peer,\n'
                        f'{ipc_err!r}'
                    )
                case _:
                    # if the feed consumer goes down then drop
                    # out of this rate limiter
                    log.warning(
                        f'{stream} closed due to,\n'
                        f'{ipc_err!r}'
                    )
            await stream.aclose()
            return
--- a/piker/data/_sharedmem.py
+++ b/piker/data/_sharedmem.py
@ -32,9 +32,21 @@ import numpy as np
 from numpy.lib import recfunctions as rfn
 import tractor
-from ._util import log
+from ..log import get_logger
-from ._source import def_iohlcv_fields
+from ._source import base_iohlc_dtype
-from piker.types import Struct
+from .types import Struct
 log = get_logger(__name__)
 # how  much is probably dependent on lifestyle
 _secs_in_day = int(60 * 60 * 24)
 # we try for a buncha times, but only on a run-every-other-day kinda week.
 _days_worth = 16
 _default_size = _days_worth * _secs_in_day
 # where to start the new data append index
 _rt_buffer_start = int((_days_worth - 1) * _secs_in_day)
 def cuckoff_mantracker():
@ -61,6 +73,7 @@ def cuckoff_mantracker():
    mantracker._resource_tracker = ManTracker()
    mantracker.register = mantracker._resource_tracker.register
    mantracker.ensure_running = mantracker._resource_tracker.ensure_running
    # ensure_running = mantracker._resource_tracker.ensure_running
    mantracker.unregister = mantracker._resource_tracker.unregister
    mantracker.getfd = mantracker._resource_tracker.getfd
@ -158,7 +171,7 @@ def _make_token(
    to access a shared array.
    '''
-    dtype = def_iohlcv_fields if dtype is None else dtype
+    dtype = base_iohlc_dtype if dtype is None else dtype
    return _Token(
        shm_name=key,
        shm_first_index_name=key + "_first",
@ -248,6 +261,7 @@ class ShmArray:
        # to load an empty array..
        if len(a) == 0 and self._post_init:
            raise RuntimeError('Empty array race condition hit!?')
            # breakpoint()
        return a
@ -257,7 +271,7 @@ class ShmArray:
        # type that all field values will be cast to
        # in the returned view.
-        common_dtype: np.dtype = float,
+        common_dtype: np.dtype = np.float,
    ) -> np.ndarray:
@ -312,7 +326,7 @@ class ShmArray:
        field_map: Optional[dict[str, str]] = None,
        prepend: bool = False,
        update_first: bool = True,
-        start: int | None = None,
+        start: Optional[int] = None,
    ) -> int:
        '''
@ -354,11 +368,7 @@ class ShmArray:
            # tries to access ``.array`` (which due to the index
            # overlap will be empty). Pretty sure we've fixed it now
            # but leaving this here as a reminder.
-            if (
+            if prepend and update_first and length:
                prepend
                and update_first
                and length
            ):
                assert index < self._first.value
            if (
@ -432,10 +442,10 @@ class ShmArray:
 def open_shm_array(
-    size: int,
+
-    key: str | None = None,
+    key: Optional[str] = None,
-    dtype: np.dtype | None = None,
+    size: int = _default_size,  # see above
-    append_start_index: int | None = None,
+    dtype: Optional[np.dtype] = None,
    readonly: bool = False,
 ) -> ShmArray:
@ -500,13 +510,10 @@ def open_shm_array(
    # ``ShmArray._start.value: int = 0`` and the yet-to-be written
    # real-time section will start at ``ShmArray.index: int``.
-    # this sets the index to nearly 2/3rds into the the length of
+    # this sets the index to 3/4 of the length of the buffer
-    # the buffer leaving at least a "days worth of second samples"
+    # leaving a "days worth of second samples" for the real-time
-    # for the real-time section.
+    # section.
-    if append_start_index is None:
+    last.value = first.value = _rt_buffer_start
        append_start_index = round(size * 0.616)
    last.value = first.value = append_start_index
    shmarr = ShmArray(
        array,
@ -520,12 +527,10 @@ def open_shm_array(
    # "unlink" created shm on process teardown by
    # pushing teardown calls onto actor context stack
-    stack = tractor.current_actor(
+
-        err_on_no_runtime=False,
+    stack = tractor.current_actor().lifetime_stack
-    ).lifetime_stack
+    stack.callback(shmarr.close)
-    if stack:
+    stack.callback(shmarr.destroy)
        stack.callback(shmarr.close)
        stack.callback(shmarr.destroy)
    return shmarr
@ -610,20 +615,14 @@ def attach_shm_array(
        _known_tokens[key] = token
    # "close" attached shm on actor teardown
-    if (actor := tractor.current_actor(
+    tractor.current_actor().lifetime_stack.callback(sha.close)
        err_on_no_runtime=False,
    )):
        actor.lifetime_stack.callback(sha.close)
    return sha
 def maybe_open_shm_array(
    key: str,
-    size: int,
+    dtype: Optional[np.dtype] = None,
    dtype: np.dtype | None = None,
    append_start_index: int | None = None,
    readonly: bool = False,
    **kwargs,
 ) -> tuple[ShmArray, bool]:
@ -644,18 +643,13 @@ def maybe_open_shm_array(
    use ``attach_shm_array``.
    '''
    size = kwargs.pop('size', _default_size)
    try:
        # see if we already know this key
        token = _known_tokens[key]
-        return (
+        return attach_shm_array(token=token, **kwargs), False
            attach_shm_array(
                token=token,
                readonly=readonly,
            ),
            False,
        )
    except KeyError:
-        log.debug(f"Could not find {key} in shms cache")
+        log.warning(f"Could not find {key} in shms cache")
        if dtype:
            token = _make_token(
                key,
@ -665,23 +659,15 @@ def maybe_open_shm_array(
            try:
                return attach_shm_array(token=token, **kwargs), False
            except FileNotFoundError:
-                log.debug(f"Could not attach to shm with token {token}")
+                log.warning(f"Could not attach to shm with token {token}")
        # This actor does not know about memory
        # associated with the provided "key".
        # Attempt to open a block and expect
        # to fail if a block has been allocated
        # on the OS by someone else.
-        return (
+        return open_shm_array(key=key, dtype=dtype, **kwargs), True
-            open_shm_array(
+
                key=key,
                size=size,
                dtype=dtype,
                append_start_index=append_start_index,
                readonly=readonly,
            ),
            True,
        )
 def try_read(
    array: np.ndarray
--- a/piker/data/_source.py
+++ b/piker/data/_source.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) 2018-present  Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) 2018-present  Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -18,47 +18,35 @@
 numpy data source coversion helpers.
 """
 from __future__ import annotations
 from typing import Any
 import decimal
 from bidict import bidict
 import numpy as np
 from .types import Struct
 # from numba import from_dtype
 def_iohlcv_fields: list[tuple[str, type]] = [
-    # YES WE KNOW, this isn't needed in polars but we use it for doing
+ohlc_fields = [
-    # ring-buffer like pre/append ops our our `ShmArray` real-time
+    ('time', float),
    # numpy-array buffering system such that there is a master index
    # that can be used for index-arithmetic when write data to the
    # "middle" of the array. See the ``tractor.ipc.shm`` pkg for more
    # details.
    ('index', int),
    # presume int for epoch stamps since it's most common
    # and makes the most sense to avoid float rounding issues.
    # TODO: if we want higher reso we should use the new
    # ``time.time_ns()`` in python 3.10+
    ('time', int),
    ('open', float),
    ('high', float),
    ('low', float),
    ('close', float),
    ('volume', float),
-
+    ('bar_wap', float),
    # TODO: can we elim this from default field set to save on mem?
    # i think only kraken really uses this in terms of what we get from
    # their ohlc history API?
    # ('bar_wap', float),  # shouldn't be default right?
 ]
-# remove index field
+ohlc_with_index = ohlc_fields.copy()
-def_ohlcv_fields: list[tuple[str, type]] = def_iohlcv_fields.copy()
+ohlc_with_index.insert(0, ('index', int))
-def_ohlcv_fields.pop(0)
+
-assert (len(def_iohlcv_fields) - len(def_ohlcv_fields)) == 1
+# our minimum structured array layout for ohlc data
 base_iohlc_dtype = np.dtype(ohlc_with_index)
 base_ohlc_dtype = np.dtype(ohlc_fields)
 # TODO: for now need to construct this manually for readonly arrays, see
 # https://github.com/numba/numba/issues/4511
 # from numba import from_dtype
 # base_ohlc_dtype = np.dtype(def_ohlc_fields)
 # numba_ohlc_dtype = from_dtype(base_ohlc_dtype)
 # map time frame "keys" to seconds values
@ -73,6 +61,28 @@ tf_in_1s = bidict({
 })
 def mk_fqsn(
    provider: str,
    symbol: str,
 ) -> str:
    '''
    Generate a "fully qualified symbol name" which is
    a reverse-hierarchical cross broker/provider symbol
    '''
    return '.'.join([symbol, provider]).lower()
 def float_digits(
    value: float,
 ) -> int:
    if value == 0:
        return 0
    return int(-decimal.Decimal(str(value)).as_tuple().exponent)
 def ohlc_zeros(length: int) -> np.ndarray:
    """Construct an OHLC field formatted structarray.
@ -83,6 +93,168 @@ def ohlc_zeros(length: int) -> np.ndarray:
    return np.zeros(length, dtype=base_ohlc_dtype)
 def unpack_fqsn(fqsn: str) -> tuple[str, str, str]:
    '''
    Unpack a fully-qualified-symbol-name to ``tuple``.
    '''
    venue = ''
    suffix = ''
    # TODO: probably reverse the order of all this XD
    tokens = fqsn.split('.')
    if len(tokens) < 3:
        # probably crypto
        symbol, broker = tokens
        return (
            broker,
            symbol,
            '',
        )
    elif len(tokens) > 3:
        symbol, venue, suffix, broker = tokens
    else:
        symbol, venue, broker = tokens
        suffix = ''
    # head, _, broker = fqsn.rpartition('.')
    # symbol, _, suffix = head.rpartition('.')
    return (
        broker,
        '.'.join([symbol, venue]),
        suffix,
    )
 class Symbol(Struct):
    '''
    I guess this is some kinda container thing for dealing with
    all the different meta-data formats from brokers?
    '''
    key: str
    tick_size: float = 0.01
    lot_tick_size: float = 0.0  # "volume" precision as min step value
    tick_size_digits: int = 2
    lot_size_digits: int = 0
    suffix: str = ''
    broker_info: dict[str, dict[str, Any]] = {}
    # specifies a "class" of financial instrument
    # ex. stock, futer, option, bond etc.
    # @validate_arguments
    @classmethod
    def from_broker_info(
        cls,
        broker: str,
        symbol: str,
        info: dict[str, Any],
        suffix: str = '',
    ) -> Symbol:
        tick_size = info.get('price_tick_size', 0.01)
        lot_tick_size = info.get('lot_tick_size', 0.0)
        return Symbol(
            key=symbol,
            tick_size=tick_size,
            lot_tick_size=lot_tick_size,
            tick_size_digits=float_digits(tick_size),
            lot_size_digits=float_digits(lot_tick_size),
            suffix=suffix,
            broker_info={broker: info},
        )
    @classmethod
    def from_fqsn(
        cls,
        fqsn: str,
        info: dict[str, Any],
    ) -> Symbol:
        broker, key, suffix = unpack_fqsn(fqsn)
        return cls.from_broker_info(
            broker,
            key,
            info=info,
            suffix=suffix,
        )
    @property
    def type_key(self) -> str:
        return list(self.broker_info.values())[0]['asset_type']
    @property
    def brokers(self) -> list[str]:
        return list(self.broker_info.keys())
    def nearest_tick(self, value: float) -> float:
        '''
        Return the nearest tick value based on mininum increment.
        '''
        mult = 1 / self.tick_size
        return round(value * mult) / mult
    def front_feed(self) -> tuple[str, str]:
        '''
        Return the "current" feed key for this symbol.
        (i.e. the broker + symbol key in a tuple).
        '''
        return (
            list(self.broker_info.keys())[0],
            self.key,
        )
    def tokens(self) -> tuple[str]:
        broker, key = self.front_feed()
        if self.suffix:
            return (key, self.suffix, broker)
        else:
            return (key, broker)
    @property
    def fqsn(self) -> str:
        return '.'.join(self.tokens()).lower()
    def front_fqsn(self) -> str:
        '''
        fqsn = "fully qualified symbol name"
        Basically the idea here is for all client-ish code (aka programs/actors
        that ask the provider agnostic layers in the stack for data) should be
        able to tell which backend / venue / derivative each data feed/flow is
        from by an explicit string key of the current form:
        <instrumentname>.<venue>.<suffixwithmetadata>.<brokerbackendname>
        TODO: I have thoughts that we should actually change this to be
        more like an "attr lookup" (like how the web should have done
        urls, but marketting peeps ruined it etc. etc.):
        <broker>.<venue>.<instrumentname>.<suffixwithmetadata>
        '''
        tokens = self.tokens()
        fqsn = '.'.join(map(str.lower, tokens))
        return fqsn
    def iterfqsns(self) -> list[str]:
        keys = []
        for broker in self.broker_info.keys():
            fqsn = mk_fqsn(self.key, broker)
            if self.suffix:
                fqsn += f'.{self.suffix}'
            keys.append(fqsn)
        return keys
 def _nan_to_closest_num(array: np.ndarray):
    """Return interpolated values instead of NaN.
--- a/piker/data/_symcache.py
+++ b/piker/data/_symcache.py
@ -1,534 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Mega-simple symbology cache via TOML files.
 Allow backend data providers and/or brokers to stash their
 symbology sets (aka the meta data we normalize into our
 `.accounting.MktPair` type) to the filesystem for faster lookup and
 offline usage.
 '''
 from __future__ import annotations
 from contextlib import (
    asynccontextmanager as acm,
 )
 from pathlib import Path
 from pprint import pformat
 from typing import (
    Any,
    Callable,
    Sequence,
    Hashable,
    TYPE_CHECKING,
 )
 from types import ModuleType
 from rapidfuzz import process as fuzzy
 import tomli_w  # for fast symbol cache writing
 import tractor
 import trio
 try:
    import tomllib
 except ModuleNotFoundError:
    import tomli as tomllib
 from msgspec import field
 from piker.log import get_logger
 from piker import config
 from piker.types import Struct
 from piker.brokers import (
    open_cached_client,
    get_brokermod,
 )
 if TYPE_CHECKING:
    from piker.accounting import (
        Asset,
        MktPair,
    )
 log = get_logger('data.cache')
 class SymbologyCache(Struct):
    '''
    Asset meta-data cache which holds lookup tables for 3 sets of
    market-symbology related struct-types required by the
    `.accounting` and `.data` subsystems.
    '''
    mod: ModuleType
    fp: Path
    # all asset-money-systems descriptions as minimally defined by
    # in `.accounting.Asset`
    assets: dict[str, Asset] = field(default_factory=dict)
    # backend-system pairs loaded in provider (schema) specific
    # structs.
    pairs: dict[str, Struct] = field(default_factory=dict)
    # serialized namespace path to the backend's pair-info-`Struct`
    # defn B)
    pair_ns_path: tractor.msg.NamespacePath | None = None
    # TODO: piker-normalized `.accounting.MktPair` table?
    # loaded from the `.pairs` and a normalizer
    # provided by the backend pkg.
    mktmaps: dict[str, MktPair] = field(default_factory=dict)
    def pformat(self) -> str:
        return (
            f'<{type(self).__name__}(\n'
            f' .mod: {self.mod!r}\n'
            f' .assets: {len(self.assets)!r}\n'
            f' .pairs: {len(self.pairs)!r}\n'
            f' .mktmaps: {len(self.mktmaps)!r}\n'
            f')>'
        )
    __repr__ = pformat
    def write_config(self) -> None:
        # put the backend's pair-struct type ref at the top
        # of file if possible.
        cachedict: dict[str, Any] = {
            'pair_ns_path': str(self.pair_ns_path) or '',
        }
        # serialize all tables as dicts for TOML.
        for key, table in {
            'assets': self.assets,
            'pairs': self.pairs,
            'mktmaps': self.mktmaps,
        }.items():
            if not table:
                log.warning(
                    f'Asset cache table for `{key}` is empty?'
                )
                continue
            dct = cachedict[key] = {}
            for key, struct in table.items():
                dct[key] = struct.to_dict(include_non_members=False)
        try:
            with self.fp.open(mode='wb') as fp:
                tomli_w.dump(cachedict, fp)
        except TypeError:
            self.fp.unlink()
            raise
    async def load(self) -> None:
        '''
        Explicitly load the "symbology set" for this provider by using
        2 required `Client` methods:
          - `.get_assets()`: returning a table of `Asset`s
          - `.get_mkt_pairs()`: returning a table of pair-`Struct`
            types, custom defined by the particular backend.
        AND, the required `.get_mkt_info()` module-level endpoint
        which maps `fqme: str` -> `MktPair`s.
        These tables are then used to fill out the `.assets`, `.pairs` and
        `.mktmaps` tables on this cache instance, respectively.
        '''
        async with open_cached_client(self.mod.name) as client:
            if get_assets := getattr(client, 'get_assets', None):
                assets: dict[str, Asset] = await get_assets()
                for bs_mktid, asset in assets.items():
                    self.assets[bs_mktid] = asset
            else:
                log.warning(
                    'No symbology cache `Asset` support for `{provider}`..\n'
                    'Implement `Client.get_assets()`!'
                )
            get_mkt_pairs: Callable|None = getattr(
                client,
                'get_mkt_pairs',
                None,
            )
            if not get_mkt_pairs:
                log.warning(
                    'No symbology cache `Pair` support for `{provider}`..\n'
                    'Implement `Client.get_mkt_pairs()`!'
                )
                return self
            pairs: dict[str, Struct] = await get_mkt_pairs()
            if not pairs:
                log.warning(
                    'No pairs from intial {provider!r} sym-cache request?\n\n'
                    '`Client.get_mkt_pairs()` -> {pairs!r} ?'
                )
                return self
            for bs_fqme, pair in pairs.items():
                if not getattr(pair, 'ns_path', None):
                    # XXX: every backend defined pair must declare
                    # a `.ns_path: tractor.NamespacePath` to enable
                    # roundtrip serialization lookup from a local
                    # cache file.
                    raise TypeError(
                        f'Pair-struct for {self.mod.name} MUST define a '
                        '`.ns_path: str`!\n\n'
                        f'{pair!r}'
                    )
                entry = await self.mod.get_mkt_info(pair.bs_fqme)
                if not entry:
                    continue
                mkt: MktPair
                pair: Struct
                mkt, _pair = entry
                assert _pair is pair, (
                    f'`{self.mod.name}` backend probably has a '
                    'keying-symmetry problem between the pair-`Struct` '
                    'returned from `Client.get_mkt_pairs()`and the '
                    'module level endpoint: `.get_mkt_info()`\n\n'
                    "Here's the struct diff:\n"
                    f'{_pair - pair}'
                )
                # NOTE XXX: this means backends MUST implement
                # a `Struct.bs_mktid: str` field to provide
                # a native-keyed map to their own symbol
                # set(s).
                self.pairs[pair.bs_mktid] = pair
                # NOTE: `MktPair`s are keyed here using piker's
                # internal FQME schema so that search,
                # accounting and feed init can be accomplished
                # a sane, uniform, normalized basis.
                self.mktmaps[mkt.fqme] = mkt
            self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
                pair,
            )
        return self
    @classmethod
    def from_dict(
        cls: type,
        data: dict,
        **kwargs,
    ) -> SymbologyCache:
        # normal init inputs
        cache = cls(**kwargs)
        # XXX WARNING: this may break if backend namespacing
        # changes (eg. `Pair` class def is moved to another
        # module) in which case you can manually update the
        # `pair_ns_path` in the symcache file and try again.
        # TODO: probably a verbose error about this?
        Pair: type = tractor.msg.NamespacePath(
            str(data['pair_ns_path'])
        ).load_ref()
        pairtable = data.pop('pairs')
        for key, pairtable in pairtable.items():
            # allow each serialized pair-dict-table to declare its
            # specific struct type's path in cases where a backend
            # supports multiples (normally with different
            # schemas..) and we are storing them in a flat `.pairs`
            # table.
            ThisPair = Pair
            if this_pair_type := pairtable.get('ns_path'):
                ThisPair: type = tractor.msg.NamespacePath(
                    str(this_pair_type)
                ).load_ref()
            pair: Struct = ThisPair(**pairtable)
            cache.pairs[key] = pair
        from ..accounting import (
            Asset,
            MktPair,
        )
        # load `dict` -> `Asset`
        assettable = data.pop('assets')
        for name, asdict in assettable.items():
            cache.assets[name] = Asset.from_msg(asdict)
        # load `dict` -> `MktPair`
        dne: list[str] = []
        mkttable = data.pop('mktmaps')
        for fqme, mktdict in mkttable.items():
            mkt = MktPair.from_msg(mktdict)
            assert mkt.fqme == fqme
            # sanity check asset refs from those (presumably)
            # loaded asset set above.
            src: Asset = cache.assets[mkt.src.name]
            assert src == mkt.src
            dst: Asset
            if not (dst := cache.assets.get(mkt.dst.name)):
                dne.append(mkt.dst.name)
                continue
            else:
                assert dst.name == mkt.dst.name
            cache.mktmaps[fqme] = mkt
        log.warning(
            f'These `MktPair.dst: Asset`s DNE says `{cache.mod.name}`?\n'
            f'{pformat(dne)}'
        )
        return cache
    @staticmethod
    async def from_scratch(
        mod: ModuleType,
        fp: Path,
        **kwargs,
    ) -> SymbologyCache:
        '''
        Generate (a) new symcache (contents) entirely from scratch
        including all (TOML) serialized data and file.
        '''
        log.info(f'GENERATING symbology cache for `{mod.name}`')
        cache = SymbologyCache(
            mod=mod,
            fp=fp,
            **kwargs,
        )
        await cache.load()
        cache.write_config()
        return cache
    def search(
        self,
        pattern: str,
        table: str = 'mktmaps'
    ) -> dict[str, Struct]:
        '''
        (Fuzzy) search this cache's `.mktmaps` table, which is
        keyed by FQMEs, for `pattern: str` and return the best
        matches in a `dict` including the `MktPair` values.
        '''
        matches = fuzzy.extract(
            pattern,
            getattr(self, table),
            score_cutoff=50,
        )
        # repack in dict[fqme, MktPair] form
        return {
            item[0].fqme: item[0]
            for item in matches
        }
 # actor-process-local in-mem-cache of symcaches (by backend).
 _caches: dict[str, SymbologyCache] = {}
 def mk_cachefile(
    provider: str,
 ) -> Path:
    cachedir: Path = config.get_conf_dir() / '_cache'
    if not cachedir.is_dir():
        log.info(f'Creating `nativedb` director: {cachedir}')
        cachedir.mkdir()
    cachefile: Path = cachedir / f'{str(provider)}.symcache.toml'
    cachefile.touch()
    return cachefile
@acm
 async def open_symcache(
    mod_or_name: ModuleType | str,
    reload: bool = False,
    only_from_memcache: bool = False,  # no API req
    _no_symcache: bool = False,  # no backend support
 ) -> SymbologyCache:
    if isinstance(mod_or_name, str):
        mod = get_brokermod(mod_or_name)
    else:
        mod: ModuleType = mod_or_name
    provider: str = mod.name
    cachefile: Path = mk_cachefile(provider)
    # NOTE: certain backends might not support a symbology cache
    # (easily) and thus we allow for an empty instance to be loaded
    # and manually filled in at the whim of the caller presuming
    # the backend pkg-module is annotated appropriately.
    if (
        getattr(mod, '_no_symcache', False)
        or _no_symcache
    ):
        yield SymbologyCache(
            mod=mod,
            fp=cachefile,
        )
        # don't do nuttin
        return
    # actor-level cache-cache XD
    global _caches
    if not reload:
        try:
            yield _caches[provider]
        except KeyError:
            msg: str = (
                f'No asset info cache exists yet for `{provider}`'
            )
            if only_from_memcache:
                raise RuntimeError(msg)
            else:
                log.warning(msg)
    # if no cache exists or an explicit reload is requested, load
    # the provider API and call appropriate endpoints to populate
    # the mkt and asset tables.
    if (
        reload
        or not cachefile.is_file()
    ):
        cache = await SymbologyCache.from_scratch(
            mod=mod,
            fp=cachefile,
        )
    else:
        log.info(
            f'Loading EXISTING `{mod.name}` symbology cache:\n'
            f'> {cachefile}'
        )
        import time
        now = time.time()
        with cachefile.open('rb') as existing_fp:
            data: dict[str, dict] = tomllib.load(existing_fp)
            log.runtime(f'SYMCACHE TOML LOAD TIME: {time.time() - now}')
            # if there's an empty file for some reason we need
            # to do a full reload as well!
            if not data:
                cache = await SymbologyCache.from_scratch(
                    mod=mod,
                    fp=cachefile,
                )
            else:
                cache = SymbologyCache.from_dict(
                    data,
                    mod=mod,
                    fp=cachefile,
                )
        # TODO: use a real profiling sys..
        # https://github.com/pikers/piker/issues/337
        log.info(f'SYMCACHE LOAD TIME: {time.time() - now}')
    yield cache
    # TODO: write only when changes detected? but that should
    # never happen right except on reload?
    # cache.write_config()
 def get_symcache(
    provider: str,
    force_reload: bool = False,
 ) -> SymbologyCache:
    '''
    Get any available symbology/assets cache from sync code by
    (maybe) manually running `trio` to do the work.
    '''
    # spawn tractor runtime and generate cache
    # if not existing.
    async def sched_gen_symcache():
        async with (
            # only for runtime's debug mode
            tractor.open_nursery(debug_mode=True),
            open_symcache(
                get_brokermod(provider),
                reload=force_reload,
            ) as symcache,
        ):
            return symcache
    try:
        symcache: SymbologyCache = trio.run(sched_gen_symcache)
        assert symcache
    except BaseException:
        import pdbp
        pdbp.xpm()
    return symcache
 def match_from_pairs(
    pairs: dict[str, Struct],
    query: str,
    score_cutoff: int = 50,
    **extract_kwargs,
 ) -> dict[str, Struct]:
    '''
    Fuzzy search over a "pairs table" maintained by most backends
    as part of their symbology-info caching internals.
    Scan the native symbol key set and return best ranked
    matches back in a new `dict`.
    '''
    # TODO: somehow cache this list (per call) like we were in
    # `open_symbol_search()`?
    keys: list[str] = list(pairs)
    matches: list[tuple[
        Sequence[Hashable],  # matching input key
        Any,  # scores
        Any,
    ]] = fuzzy.extract(
        # NOTE: most backends provide keys uppercased
        query=query,
        choices=keys,
        score_cutoff=score_cutoff,
        **extract_kwargs,
    )
    # pop and repack pairs in output dict
    matched_pairs: dict[str, Struct] = {}
    for item in matches:
        pair_key: str = item[0]
        matched_pairs[pair_key] = pairs[pair_key]
    return matched_pairs
--- a/piker/data/_util.py
+++ b/piker/data/_util.py
@ -1,36 +0,0 @@
 # piker: trading gear for hackers
 # Copyright (C) Tyler Goodlet (in stewardship for pikers)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 '''
 Data layer module commons.
 '''
 from functools import partial
 from ..log import (
    get_logger,
    get_console_log,
 )
 subsys: str = 'piker.data'
 log = get_logger(
    name=subsys,
 )
 get_console_log = partial(
    get_console_log,
    name=subsys,
 )
--- a/piker/data/_web_bs.py
+++ b/piker/data/_web_bs.py
@ -1,5 +1,5 @@
 # piker: trading gear for hackers
-# Copyright (C) Tyler Goodlet (in stewardship for pikers)
+# Copyright (C) Tyler Goodlet (in stewardship for piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@ -18,30 +18,23 @@
 ToOlS fOr CoPInG wITh "tHE wEB" protocols.
 """
 from __future__ import annotations
 from contextlib import (
-    asynccontextmanager as acm,
+    asynccontextmanager,
    AsyncExitStack,
 )
 from itertools import count
 from functools import partial
 from types import ModuleType
 from typing import (
    Any,
    Optional,
    Callable,
    AsyncContextManager,
    AsyncGenerator,
    Iterable,
    Type,
 )
 import json
 import tractor
 import trio
-from trio_typing import TaskStatus
+import trio_websocket
 from trio_websocket import (
    WebSocketConnection,
    open_websocket_url,
 )
 from wsproto.utilities import LocalProtocolError
 from trio_websocket._impl import (
    ConnectionClosed,
@ -51,24 +44,21 @@ from trio_websocket._impl import (
    ConnectionTimeout,
 )
-from piker.types import Struct
+from ..log import get_logger
-from ._util import log
+
 from .types import Struct
 log = get_logger(__name__)
 class NoBsWs:
    '''
    Make ``trio_websocket`` sockets stay up no matter the bs.
-    A shim interface that allows client code to stream from some
+    You can provide a ``fixture`` async-context-manager which will be
-    ``WebSocketConnection`` but where any connectivy bs is handled
+    enter/exitted around each reconnect operation.
    automatcially and entirely in the background.
    NOTE: this type should never be created directly but instead is
    provided via the ``open_autorecon_ws()`` factor below.
    '''
-    # apparently we can QoS for all sorts of reasons..so catch em.
+    recon_errors = (
    recon_errors: tuple[Type[Exception]] = (
        ConnectionClosed,
        DisconnectionTimeout,
        ConnectionRejected,
@ -80,74 +70,87 @@ class NoBsWs:
    def __init__(
        self,
        url: str,
-        rxchan: trio.MemoryReceiveChannel,
+        stack: AsyncExitStack,
-        msg_recv_timeout: float,
+        fixture: Optional[Callable] = None,
        serializer: ModuleType = json
    ):
        self.url = url
-        self._rx = rxchan
+        self.fixture = fixture
-        self._timeout = msg_recv_timeout
+        self._stack = stack
        self._ws: 'WebSocketConnection' = None  # noqa
-        # signaling between caller and relay task which determines when
+        # TODO: is there some method we can call
-        # socket is connected (and subscribed).
+        # on the underlying `._ws` to get this?
-        self._connected: trio.Event = trio.Event()
+        self._connected: bool = False
-        # dynamically reset by the bg relay task
+    async def _connect(
-        self._ws: WebSocketConnection | None = None
+        self,
-        self._cs: trio.CancelScope | None = None
+        tries: int = 1000,
    ) -> None:
-        # interchange codec methods
+        self._connected = False
-        # TODO: obviously the method API here may be different
+        while True:
-        # for another interchange format..
+            try:
-        self._dumps: Callable = serializer.dumps
+                await self._stack.aclose()
-        self._loads: Callable = serializer.loads
+            except self.recon_errors:
                await trio.sleep(0.5)
            else:
                break
        last_err = None
        for i in range(tries):
            try:
                self._ws = await self._stack.enter_async_context(
                    trio_websocket.open_websocket_url(self.url)
                )
                if self.fixture is not None:
                    # rerun user code fixture
                    ret = await self._stack.enter_async_context(
                        self.fixture(self)
                    )
                    assert ret is None
                log.info(f'Connection success: {self.url}')
                self._connected = True
                return self._ws
            except self.recon_errors as err:
                last_err = err
                log.error(
                    f'{self} connection bail with '
                    f'{type(err)}...retry attempt {i}'
                )
                await trio.sleep(0.5)
                self._connected = False
                continue
        else:
            log.exception('ws connection fail...')
            raise last_err
    def connected(self) -> bool:
-        return self._connected.is_set()
+        return self._connected
    async def reset(
        self,
        timeout: float,
    ) -> bool:
        '''
        Reset the underlying ws connection by cancelling
        the bg relay task and waiting for it to signal
        a new connection.
        '''
        self._connected = trio.Event()
        self._cs.cancel()
        with trio.move_on_after(timeout) as cs:
            await self._connected.wait()
            return True
        assert cs.cancelled_caught
        return False
    async def send_msg(
        self,
        data: Any,
        timeout: float = 3,
    ) -> None:
        while True:
            try:
-                msg: Any = self._dumps(data)
+                return await self._ws.send_message(json.dumps(data))
                return await self._ws.send_message(msg)
            except self.recon_errors:
-                with trio.CancelScope(shield=True):
+                await self._connect()
                    reconnected: bool = await self.reset(
                        timeout=timeout,
                    )
                    if not reconnected:
                        log.warning(
                            'Failed to reconnect after {timeout!r}s ??'
                        )
-    async def recv_msg(self) -> Any:
+    async def recv_msg(
-        msg: Any = await self._rx.receive()
+        self,
-        data = self._loads(msg)
+    ) -> Any:
-        return data
+        while True:
            try:
                return json.loads(await self._ws.get_message())
            except self.recon_errors:
                await self._connect()
    def __aiter__(self):
        return self
@ -155,260 +158,32 @@ class NoBsWs:
    async def __anext__(self):
        return await self.recv_msg()
    def set_recv_timeout(
        self,
        timeout: float,
    ) -> None:
        self._timeout = timeout
-
+@asynccontextmanager
 async def _reconnect_forever(
    url: str,
    snd: trio.MemorySendChannel,
    nobsws: NoBsWs,
    reset_after: int,  # msg recv timeout before reset attempt
    fixture: AsyncContextManager | None = None,
    task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
 ) -> None:
    # TODO: can we just report "where" in the call stack
    # the client code is using the ws stream?
    # Maybe we can just drop this since it's already in the log msg
    # orefix?
    if fixture is not None:
        src_mod: str = fixture.__module__
    else:
        src_mod: str = 'unknown'
    async def proxy_msgs(
        ws: WebSocketConnection,
        rent_cs: trio.CancelScope,  # parent cancel scope
    ):
        '''
        Receive (under `timeout` deadline) all msgs from from underlying
        websocket and relay them to (calling) parent task via ``trio``
        mem chan.
        '''
        # after so many msg recv timeouts, reset the connection
        timeouts: int = 0
        while True:
            with trio.move_on_after(
                # can be dynamically changed by user code
                nobsws._timeout,
            ) as cs:
                try:
                    msg: Any = await ws.get_message()
                    await snd.send(msg)
                except nobsws.recon_errors:
                    log.exception(
                        f'{src_mod}\n'
                        f'{url} connection bail with:'
                    )
                    with trio.CancelScope(shield=True):
                        await trio.sleep(0.5)
                    rent_cs.cancel()
                    # go back to reonnect loop in parent task
                    return
            if cs.cancelled_caught:
                timeouts += 1
                if timeouts > reset_after:
                    log.error(
                        f'{src_mod}\n'
                        'WS feed seems down and slow af.. reconnecting\n'
                    )
                    rent_cs.cancel()
                    # go back to reonnect loop in parent task
                    return
    async def open_fixture(
        fixture: AsyncContextManager,
        nobsws: NoBsWs,
        task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
    ):
        '''
        Open user provided `@acm` and sleep until any connection
        reset occurs.
        '''
        async with fixture(nobsws) as ret:
            assert ret is None
            task_status.started()
            await trio.sleep_forever()
    # last_err = None
    nobsws._connected = trio.Event()
    task_status.started()
    mc_state: trio._channel.MemoryChannelState = snd._state
    while (
        mc_state.open_receive_channels > 0
        and
        mc_state.open_send_channels > 0
    ):
        log.info(
            f'{src_mod}\n'
            f'{url} trying (RE)CONNECT'
        )
        ws: WebSocketConnection
        try:
            async with (
                open_websocket_url(url) as ws,
                tractor.trionics.collapse_eg(),
                trio.open_nursery() as tn,
            ):
                cs = nobsws._cs = tn.cancel_scope
                nobsws._ws = ws
                log.info(
                    f'{src_mod}\n'
                    f'Connection success: {url}'
                )
                # begin relay loop to forward msgs
                tn.start_soon(
                    proxy_msgs,
                    ws,
                    cs,
                )
                if fixture is not None:
                    log.info(
                        f'{src_mod}\n'
                        f'Entering fixture: {fixture}'
                    )
                    # TODO: should we return an explicit sub-cs
                    # from this fixture task?
                    await tn.start(
                        open_fixture,
                        fixture,
                        nobsws,
                    )
                # indicate to wrapper / opener that we are up and block
                # to let tasks run **inside** the ws open block above.
                nobsws._connected.set()
                await trio.sleep_forever()
        except (
            HandshakeError,
            ConnectionRejected,
        ):
            log.exception('Retrying connection')
            await trio.sleep(0.5)  # throttle
        except BaseException as _berr:
            berr = _berr
            log.exception(
                'Reconnect-attempt failed ??\n'
            )
            with trio.CancelScope(shield=True):
                await trio.sleep(0.2)  # throttle
            raise berr
        #|_ws & nursery block ends
        nobsws._connected = trio.Event()
        if cs.cancelled_caught:
            log.cancel(
                f'{url} connection cancelled!'
            )
            # if wrapper cancelled us, we expect it to also
            # have re-assigned a new event
            assert (
                nobsws._connected
                and not nobsws._connected.is_set()
            )
        # -> from here, move to next reconnect attempt iteration
        # in the while loop above Bp
    else:
        log.exception(
            f'{src_mod}\n'
            'ws connection closed by client...'
        )
@acm
 async def open_autorecon_ws(
    url: str,
-    fixture: AsyncContextManager | None = None,
+    # TODO: proper type cannot smh
-
+    fixture: Optional[Callable] = None,
    # time in sec between msgs received before
    # we presume connection might need a reset.
    msg_recv_timeout: float = 16,
    # count of the number of above timeouts before connection reset
    reset_after: int = 3,
 ) -> AsyncGenerator[tuple[...],  NoBsWs]:
-    '''
+    """Apparently we can QoS for all sorts of reasons..so catch em.
    An auto-reconnect websocket (wrapper API) around
    ``trio_websocket.open_websocket_url()`` providing automatic
    re-connection on network errors, msg latency and thus roaming.
-    Here we implement a re-connect websocket interface where a bg
+    """
-    nursery runs ``WebSocketConnection.receive_message()``s in a loop
+    async with AsyncExitStack() as stack:
-    and restarts the full http(s) handshake on catches of certain
+        ws = NoBsWs(url, stack, fixture=fixture)
-    connetivity errors, or some user defined recv timeout.
+        await ws._connect()
-    You can provide a ``fixture`` async-context-manager which will be
+        try:
-    entered/exitted around each connection reset; eg. for
+            yield ws
    (re)requesting subscriptions without requiring streaming setup
    code to rerun.
-    '''
+        finally:
-    snd: trio.MemorySendChannel
+            await stack.aclose()
    rcv: trio.MemoryReceiveChannel
    snd, rcv = trio.open_memory_channel(616)
    try:
        async with (
            tractor.trionics.collapse_eg(),
            trio.open_nursery() as tn
        ):
            nobsws = NoBsWs(
                url,
                rcv,
                msg_recv_timeout=msg_recv_timeout,
            )
            await tn.start(
                partial(
                    _reconnect_forever,
                    url,
                    snd,
                    nobsws,
                    fixture=fixture,
                    reset_after=reset_after,
                )
            )
            await nobsws._connected.wait()
            assert nobsws._cs
            assert nobsws.connected()
            try:
                yield nobsws
            finally:
                tn.cancel_scope.cancel()
    except NoBsWs.recon_errors as con_err:
        log.warning(
            f'Entire ws-channel disconnect due to,\n'
            f'con_err: {con_err!r}\n'
        )
 '''
-JSONRPC response-request style machinery for transparent multiplexing
+JSONRPC response-request style machinery for transparent multiplexing of msgs
-of msgs over a `NoBsWs`.
+over a NoBsWs.
 '''
@ -416,91 +191,52 @@ of msgs over a `NoBsWs`.
 class JSONRPCResult(Struct):
    id: int
    jsonrpc: str = '2.0'
-    result: dict|None = None
+    result: Optional[dict] = None
-    error: dict|None = None
+    error: Optional[dict] = None
-@acm
+@asynccontextmanager
 async def open_jsonrpc_session(
    url: str,
    start_id: int = 0,
    response_type: type = JSONRPCResult,
-    msg_recv_timeout: float = float('inf'),
+    request_type: Optional[type] = None,
-    # ^NOTE, since only `deribit` is using this jsonrpc stuff atm
+    request_hook: Optional[Callable] = None,
-    # and options mkts are generally "slow moving"..
+    error_hook: Optional[Callable] = None,
    #
    # FURTHER if we break the underlying ws connection then since we
    # don't pass a `fixture` to the task that manages `NoBsWs`, i.e.
    # `_reconnect_forever()`, the jsonrpc "transport pipe" get's
    # broken and never restored with wtv init sequence is required to
    # re-establish a working req-resp session.
 ) -> Callable[[str, dict], dict]:
    '''
    Init a json-RPC-over-websocket connection to the provided `url`.
    A `json_rpc: Callable[[str, dict], dict` is delivered to the
    caller for sending requests and a bg-`trio.Task` handles
    processing of response msgs including error reporting/raising in
    the parent/caller task.
    '''
    # NOTE, store all request msgs so we can raise errors on the
    # caller side!
    req_msgs: dict[int, dict] = {}
    async with (
-        trio.open_nursery() as tn,
+        trio.open_nursery() as n,
-        open_autorecon_ws(
+        open_autorecon_ws(url) as ws
            url=url,
            msg_recv_timeout=msg_recv_timeout,
        ) as ws
    ):
-        rpc_id: Iterable[int] = count(start_id)
+        rpc_id: Iterable = count(start_id)
        rpc_results: dict[int, dict] = {}
-        async def json_rpc(
+        async def json_rpc(method: str, params: dict) -> dict:
            method: str,
            params: dict,
        ) -> dict:
            '''
            perform a json rpc call and wait for the result, raise exception in
            case of error field present on response
            '''
            nonlocal req_msgs
            req_id: int = next(rpc_id)
            msg = {
                'jsonrpc': '2.0',
-                'id': req_id,
+                'id': next(rpc_id),
                'method': method,
                'params': params
            }
            _id = msg['id']
-            result = rpc_results[_id] = {
+            rpc_results[_id] = {
                'result': None,
-                'error': None,
+                'event': trio.Event()
                'event': trio.Event(),  # signal caller resp arrived
            }
            req_msgs[_id] = msg
            await ws.send_msg(msg)
            # wait for reponse before unblocking requester code
            await rpc_results[_id]['event'].wait()
-            if (maybe_result := result['result']):
+            ret = rpc_results[_id]['result']
                ret = maybe_result
                del rpc_results[_id]
-            else:
+            del rpc_results[_id]
                err = result['error']
                raise Exception(
                    f'JSONRPC request failed\n'
                    f'req: {msg}\n'
                    f'resp: {err}\n'
                )
            if ret.error is not None:
                raise Exception(json.dumps(ret.error, indent=4))
@ -515,7 +251,6 @@ async def open_jsonrpc_session(
            the server side.
            '''
            nonlocal req_msgs
            async for msg in ws:
                match msg:
                    case {
@ -539,28 +274,19 @@ async def open_jsonrpc_session(
                        'params': _,
                    }:
                        log.debug(f'Recieved\n{msg}')
                        if request_hook:
                            await request_hook(request_type(**msg))
                    case {
                        'error': error
                    }:
-                        # retreive orig request msg, set error
+                        log.warning(f'Recieved\n{error}')
-                        # response in original "result" msg,
+                        if error_hook:
-                        # THEN FINALLY set the event to signal caller
+                            await error_hook(response_type(**msg))
                        # to raise the error in the parent task.
                        req_id: int = error['id']
                        req_msg: dict = req_msgs[req_id]
                        result: dict = rpc_results[req_id]
                        result['error'] = error
                        result['event'].set()
                        log.error(
                            f'JSONRPC request failed\n'
                            f'req: {req_msg}\n'
                            f'resp: {error}\n'
                        )
                    case _:
                        log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
-        tn.start_soon(recv_task)
+        n.start_soon(recv_task)
        yield json_rpc
-        tn.cancel_scope.cancel()
+        n.cancel_scope.cancel()
--- a/piker/data/cli.py
+++ b/piker/data/cli.py
@ -0,0 +1,196 @@
 # piker: trading gear for hackers
 # Copyright (C) 2018-present  Tyler Goodlet (in stewardship of piker0)
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU Affero General Public License for more details.
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 """
 marketstore cli.
 """
 from functools import partial
 from pprint import pformat
 from anyio_marketstore import open_marketstore_client
 import trio
 import tractor
 import click
 import numpy as np
 from .marketstore import (
    get_client,
    # stream_quotes,
    ingest_quote_stream,
    # _url,
    _tick_tbk_ids,
    mk_tbk,
 )
 from ..cli import cli
 from .. import watchlists as wl
 from ..log import get_logger
 from ._sharedmem import (
    maybe_open_shm_array,
 )
 from ._source import (
    base_iohlc_dtype,
 )
 log = get_logger(__name__)
@cli.command()
@click.option(
    '--url',
    default='ws://localhost:5993/ws',
    help='HTTP URL of marketstore instance'
 )
@click.argument('names', nargs=-1)
@click.pass_obj
 def ms_stream(
    config: dict,
    names: list[str],
    url: str,
 ) -> None:
    '''
    Connect to a marketstore time bucket stream for (a set of) symbols(s)
    and print to console.
    '''
    async def main():
        # async for quote in stream_quotes(symbols=names):
        #    log.info(f"Received quote:\n{quote}")
        ...
    trio.run(main)
 # @cli.command()
 # @click.option(
 #     '--url',
 #     default=_url,
 #     help='HTTP URL of marketstore instance'
 # )
 # @click.argument('names', nargs=-1)
 # @click.pass_obj
 # def ms_destroy(config: dict, names: list[str], url: str) -> None:
 #     """Destroy symbol entries in the local marketstore instance.
 #     """
 #     async def main():
 #         nonlocal names
 #         async with get_client(url) as client:
 # 
 #             if not names:
 #                 names = await client.list_symbols()
 # 
 #             # default is to wipe db entirely.
 #             answer = input(
 #                 "This will entirely wipe you local marketstore db @ "
 #                 f"{url} of the following symbols:\n {pformat(names)}"
 #                 "\n\nDelete [N/y]?\n")
 # 
 #             if answer == 'y':
 #                 for sym in names:
 #                     # tbk = _tick_tbk.format(sym)
 #                     tbk = tuple(sym, *_tick_tbk_ids)
 #                     print(f"Destroying {tbk}..")
 #                     await client.destroy(mk_tbk(tbk))
 #             else:
 #                 print("Nothing deleted.")
 # 
 #     tractor.run(main)
@cli.command()
@click.option(
    '--tl',
    is_flag=True,
    help='Enable tractor logging')
@click.option(
    '--host',
    default='localhost'
 )
@click.option(
    '--port',
    default=5993
 )
@click.argument('symbols', nargs=-1)
@click.pass_obj
 def storesh(
    config,
    tl,
    host,
    port,
    symbols: list[str],
 ):
    '''
    Start an IPython shell ready to query the local marketstore db.
    '''
    from piker.data.marketstore import tsdb_history_update
    from piker._daemon import open_piker_runtime
    async def main():
        nonlocal symbols
        async with open_piker_runtime(
            'storesh',
            enable_modules=['piker.data._ahab'],
        ):
            symbol = symbols[0]
            await tsdb_history_update(symbol)
    trio.run(main)
@cli.command()
@click.option('--test-file', '-t', help='Test quote stream file')
@click.option('--tl', is_flag=True, help='Enable tractor logging')
@click.argument('name', nargs=1, required=True)
@click.pass_obj
 def ingest(config, name, test_file, tl):
    '''
    Ingest real-time broker quotes and ticks to a marketstore instance.
    '''
    # global opts
    loglevel = config['loglevel']
    tractorloglevel = config['tractorloglevel']
    # log = config['log']
    watchlist_from_file = wl.ensure_watchlists(config['wl_path'])
    watchlists = wl.merge_watchlist(watchlist_from_file, wl._builtins)
    symbols = watchlists[name]
    grouped_syms = {}
    for sym in symbols:
        symbol, _, provider = sym.rpartition('.')
        if provider not in grouped_syms:
            grouped_syms[provider] = []
        grouped_syms[provider].append(symbol)
    async def entry_point():
        async with tractor.open_nursery() as n:
            for provider, symbols in grouped_syms.items(): 
                await n.run_in_actor(
                    ingest_quote_stream,
                    name='ingest_marketstore',
                    symbols=symbols,
                    brokername=provider,
                    tries=1,
                    actorloglevel=loglevel,
                    loglevel=tractorloglevel
                )
    tractor.run(entry_point)
--- a/piker/data/feed.py
+++ b/piker/data/feed.py
--- a/Show More
+++ b/Show More