Compare commits
No commits in common. "backfiller_deep_fixes" and "main" have entirely different histories.
backfiller
...
main
|
|
@ -3,8 +3,7 @@
|
||||||
"allow": [
|
"allow": [
|
||||||
"Bash(chmod:*)",
|
"Bash(chmod:*)",
|
||||||
"Bash(/tmp/piker_commits.txt)",
|
"Bash(/tmp/piker_commits.txt)",
|
||||||
"Bash(python:*)",
|
"Bash(python:*)"
|
||||||
"Bash(ls:*)"
|
|
||||||
],
|
],
|
||||||
"deny": [],
|
"deny": [],
|
||||||
"ask": []
|
"ask": []
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,84 @@
|
||||||
|
---
|
||||||
|
name: commit-msg
|
||||||
|
description: >
|
||||||
|
Generate piker-style git commit messages from
|
||||||
|
staged changes or prompt input, following the
|
||||||
|
style guide learned from 500 repo commits.
|
||||||
|
argument-hint: "[optional-scope-or-description]"
|
||||||
|
disable-model-invocation: true
|
||||||
|
allowed-tools: Bash(git *), Read, Grep, Glob, Write
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current staged changes
|
||||||
|
!`git diff --staged --stat`
|
||||||
|
|
||||||
|
## Recent commit style reference
|
||||||
|
!`git log --oneline -10`
|
||||||
|
|
||||||
|
# Piker Git Commit Message Generator
|
||||||
|
|
||||||
|
Generate a commit message from the staged diff above
|
||||||
|
following the piker project's conventions (learned from
|
||||||
|
analyzing 500 repo commits).
|
||||||
|
|
||||||
|
If `$ARGUMENTS` is provided, use it as scope or
|
||||||
|
description context for the commit message.
|
||||||
|
|
||||||
|
For the full style guide with verb frequencies,
|
||||||
|
section markers, abbreviations, piker-specific terms,
|
||||||
|
and examples, see
|
||||||
|
[style-guide-reference.md](./style-guide-reference.md).
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
- **Subject**: ~50 chars, present tense verb, use
|
||||||
|
backticks for code refs
|
||||||
|
- **Body**: only for complex/multi-file changes,
|
||||||
|
67 char line max
|
||||||
|
- **Section markers**: Also, / Deats, / Other,
|
||||||
|
- **Bullets**: use `-` style
|
||||||
|
- **Tone**: technical but casual (piker style)
|
||||||
|
|
||||||
|
## Claude-code Footer
|
||||||
|
|
||||||
|
When the written **patch** was assisted by
|
||||||
|
claude-code, include:
|
||||||
|
|
||||||
|
```
|
||||||
|
(this patch was generated in some part by [`claude-code`][claude-code-gh])
|
||||||
|
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||||
|
```
|
||||||
|
|
||||||
|
When only the **commit msg** was written by
|
||||||
|
claude-code (human wrote the patch), use:
|
||||||
|
```
|
||||||
|
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
|
||||||
|
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Instructions
|
||||||
|
|
||||||
|
When generating a commit message:
|
||||||
|
|
||||||
|
1. Analyze the staged diff (injected above via
|
||||||
|
dynamic context) to understand all changes.
|
||||||
|
2. If `$ARGUMENTS` provides a scope (e.g.,
|
||||||
|
`.ib.feed`) or description, incorporate it into
|
||||||
|
the subject line.
|
||||||
|
3. Write the subject line following verb + backtick
|
||||||
|
conventions from the
|
||||||
|
[style guide](./style-guide-reference.md).
|
||||||
|
4. Add body only for multi-file or complex changes.
|
||||||
|
5. Write the message to a file in the repo's
|
||||||
|
`.claude/` subdir with filename format:
|
||||||
|
`<timestamp>_<first-7-chars-of-last-commit-hash>_commit_msg.md`
|
||||||
|
where `<timestamp>` is from `date --iso-8601=seconds`.
|
||||||
|
Also write a copy to
|
||||||
|
`.claude/git_commit_msg_LATEST.md`
|
||||||
|
(overwrite if exists).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Analysis date:** 2026-01-27
|
||||||
|
**Commits analyzed:** 500 from piker repository
|
||||||
|
**Maintained by:** Tyler Goodlet
|
||||||
|
|
@ -150,18 +150,21 @@ Common in piker commits (33.0% use colons):
|
||||||
- File:line references not used (0 occurrences)
|
- File:line references not used (0 occurrences)
|
||||||
- No WIP commits in analyzed set
|
- No WIP commits in analyzed set
|
||||||
|
|
||||||
### Coding-harness Footer
|
### Claude-code Footer
|
||||||
When a coding harness assisted with the written patch,
|
When the written **patch** was assisted by claude-code,
|
||||||
identify the active harness, model, and provider:
|
include:
|
||||||
|
|
||||||
```
|
```
|
||||||
(this patch was generated in some part by `<harness>` using `<model>` (`<provider>`))
|
(this patch was generated in some part by [`claude-code`][claude-code-gh])
|
||||||
|
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||||
```
|
```
|
||||||
|
|
||||||
When it generated only the commit message, use:
|
When only the **commit msg** was written by claude-code
|
||||||
|
(human wrote the patch), use:
|
||||||
|
|
||||||
```
|
```
|
||||||
(this commit msg was generated in some part by `<harness>` using `<model>` (`<provider>`))
|
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
|
||||||
|
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||||
```
|
```
|
||||||
|
|
||||||
## Piker-Specific Terms
|
## Piker-Specific Terms
|
||||||
|
|
|
||||||
|
|
@ -1,155 +0,0 @@
|
||||||
---
|
|
||||||
name: piker-conc-expert
|
|
||||||
description: >
|
|
||||||
Distributed-runtime and structured-concurrency
|
|
||||||
expertise for piker's `tractor` actor-tree. Apply
|
|
||||||
when working on daemon/service architecture, actor
|
|
||||||
spawning/discovery, cross-actor RPC (ctx/stream
|
|
||||||
eps), `to_asyncio` integration, cancellation
|
|
||||||
semantics, or debugging hangs/wedges/skews in the
|
|
||||||
actor system.
|
|
||||||
compatibility: >
|
|
||||||
Requires a piker checkout and familiarity with Python,
|
|
||||||
trio, and tractor.
|
|
||||||
metadata:
|
|
||||||
author: goodboy
|
|
||||||
version: "1.0"
|
|
||||||
---
|
|
||||||
|
|
||||||
# Piker Concurrency & Runtime Expertise
|
|
||||||
|
|
||||||
The distilled mental model for piker's distributed
|
|
||||||
runtime: a `trio`-structured actor tree supervised by
|
|
||||||
`tractor` (pinned to git main) where every long-lived
|
|
||||||
subsystem is a named daemon-actor talking over
|
|
||||||
ctx/stream IPC.
|
|
||||||
|
|
||||||
## Actor tree & daemon taxonomy
|
|
||||||
|
|
||||||
```
|
|
||||||
pikerd root supervisor + registry
|
|
||||||
├── datad.<broker> feed bus, shm writers, tsp
|
|
||||||
│ history, symbol search
|
|
||||||
├── brokerd.<broker> live order-ctl ONLY; lazily
|
|
||||||
│ spawned by emsd, credentialed
|
|
||||||
├── emsd dark-clearing + order routing
|
|
||||||
│ └── paperboi.<broker> sim-clearing (paper mode)
|
|
||||||
└── samplerd singleton OHLC clock/increment
|
|
||||||
```
|
|
||||||
|
|
||||||
Key invariants:
|
|
||||||
- `datad` hosts all `piker.data.validate._eps['datad']`
|
|
||||||
eps; `brokerd` only the `['brokerd']` (order-ctl)
|
|
||||||
ones. The `_eps` table in `piker/data/validate.py`
|
|
||||||
is the authoritative contract; `get_eps(mod, kind)`
|
|
||||||
introspects a backend's support.
|
|
||||||
- `brokerd.<broker>` is booted in EXACTLY one place:
|
|
||||||
`open_brokerd_dialog()` in `piker/clearing/_ems.py`
|
|
||||||
(with a `portal:` override for the `piker ledger`
|
|
||||||
ad-hoc actor). Chart-only + paper sessions run with
|
|
||||||
ZERO brokerd procs. Never add a data-path spawn!
|
|
||||||
- backends declare per-daemon-kind submods via
|
|
||||||
`_datad_mods`/`_brokerd_mods` in their
|
|
||||||
`__init__.py` (fallback: `__enable_modules__`).
|
|
||||||
|
|
||||||
## Daemon lifecycle conventions
|
|
||||||
|
|
||||||
Every daemon-kind follows the same trio of fns (see
|
|
||||||
`piker/brokers/_daemon.py` + `piker/data/_daemon.py`
|
|
||||||
as the canonical pair):
|
|
||||||
|
|
||||||
- `_setup_persistent_<kind>()`: a `@tractor.context`
|
|
||||||
"lifetime fixture" run via
|
|
||||||
`Services.start_service_task()`; does console-log
|
|
||||||
setup ONCE for the actor, allocs any actor-global
|
|
||||||
state (eg. datad's `_FeedsBus`), then
|
|
||||||
`await ctx.started()` + `trio.sleep_forever()`.
|
|
||||||
- `<kind>_init()`: builds `enable_modules` + actor
|
|
||||||
name `f'<kind>.{brokername}'` and copies backend
|
|
||||||
`_spawn_kwargs` (CRITICAL: `ib` needs
|
|
||||||
`infect_asyncio=True` in EVERY daemon-kind).
|
|
||||||
- `spawn_<kind>()` + `maybe_spawn_<kind>()`: thin
|
|
||||||
wrappers over `Services.actor_n.start_actor()` and
|
|
||||||
`piker.service.maybe_spawn_daemon()` (registry
|
|
||||||
find-or-spawn w/ per-service-name locking).
|
|
||||||
|
|
||||||
Caps-sec model: `enable_modules` gates RPC entry ONLY
|
|
||||||
— python imports are unrestricted in-proc. Keep each
|
|
||||||
daemon's enable set minimal; the (credentialed)
|
|
||||||
`brokerd` must never RPC-enable `piker.data.*` feed
|
|
||||||
mods.
|
|
||||||
|
|
||||||
## Actor-local state: the #1 split hazard
|
|
||||||
|
|
||||||
Module-globals and instance caches are PER-ACTOR.
|
|
||||||
Anything that "just worked" because two subsystems
|
|
||||||
shared a process will break when they're split into
|
|
||||||
sibling actors. Canonical example: `ib`'s
|
|
||||||
`Client._contracts` was warmed by feed-side
|
|
||||||
`get_mkt_info()` in-proc; post datad/brokerd-split
|
|
||||||
the trading actor must warm it itself (eagerly at
|
|
||||||
`open_trade_dialog()` startup for open pps/orders +
|
|
||||||
lazily per order request via
|
|
||||||
`symbols.cache_contract()`).
|
|
||||||
|
|
||||||
When moving code across actor boundaries ALWAYS audit:
|
|
||||||
- module-global registries (`feed._bus`,
|
|
||||||
`_accounts2clients`, `_client_cache`, ..)
|
|
||||||
- `@async_lifo_cache`/`maybe_open_context` caches
|
|
||||||
(NOTE: `async_lifo_cache` keys on POSITIONAL args
|
|
||||||
only; a cache-hit SKIPS the fn body and thus any
|
|
||||||
side-effect writes!)
|
|
||||||
- logging handler placement (see gotchas.md)
|
|
||||||
|
|
||||||
## tractor primitives as used here
|
|
||||||
|
|
||||||
- `@tractor.context` eps: `await ctx.started(val)`
|
|
||||||
unblocks the caller w/ `val`; long-lived eps then
|
|
||||||
`ctx.open_stream()` or `sleep_forever()`.
|
|
||||||
- discovery: `tractor.find_actor()` via
|
|
||||||
`piker.service.find_service()`;
|
|
||||||
`wait_for_actor(name, registry_addr=...)`;
|
|
||||||
`query_actor(name, regaddr=...)` yields
|
|
||||||
`(sockaddr, portal)`. Addrs are wrapped
|
|
||||||
`tractor.discovery._addr.Address` types — use
|
|
||||||
`wrap_address()` to normalize raw tuples and
|
|
||||||
`.unwrap()` for comparisons.
|
|
||||||
- runtime-vars: `_runtime_vars['piker_vars']` is
|
|
||||||
inherited down the spawn tree; used eg. for
|
|
||||||
`piker_test_dir` config isolation — read LAZILY at
|
|
||||||
use-time, never at import time (subactors only get
|
|
||||||
vars post runtime-boot).
|
|
||||||
- cancellation semantics (modern tractor): a
|
|
||||||
`ContextCancelled` whose `.canceller` is your own
|
|
||||||
actor is ABSORBED (clean exit, nothing raised);
|
|
||||||
single-exc groups collapse (`collapse_eg`) so eg.
|
|
||||||
a KBI propagates bare. Exc attrs:
|
|
||||||
`RemoteActorError.boxed_type` (not `.type`).
|
|
||||||
|
|
||||||
## `to_asyncio` (infect-asyncio) integration
|
|
||||||
|
|
||||||
For `ib` (and `deribit`) the backend client runs on
|
|
||||||
an embedded `asyncio` loop via
|
|
||||||
`tractor.to_asyncio.open_channel_from()` +
|
|
||||||
`LinkedTaskChannel`.
|
|
||||||
|
|
||||||
Rules learned the hard way:
|
|
||||||
- a shared req/resp channel MUST correlate responses
|
|
||||||
to requests (see `MethodProxy._run_method()`'s
|
|
||||||
`mid` protocol in `piker/brokers/ib/api.py`):
|
|
||||||
caller cancellation (eg. `move_on_after` timeouts)
|
|
||||||
otherwise orphans a response and silently skews
|
|
||||||
every later result off-by-one.
|
|
||||||
- the aio-side relay must catch + ship back ALL
|
|
||||||
(non-cancel) exceptions as `{'exception': err}`
|
|
||||||
resps; an escaping error kills the relay task ->
|
|
||||||
channel -> proxy nursery -> the whole dialog,
|
|
||||||
bypassing every caller-side guard.
|
|
||||||
- `TrioTaskExited` ("child asyncio task is still
|
|
||||||
running?") on teardown is a known wart family;
|
|
||||||
prefer upstream `tractor` fixes over piker-side
|
|
||||||
bandaids.
|
|
||||||
|
|
||||||
See [gotchas.md](gotchas.md) for the symptom->cause
|
|
||||||
registry and [debug-recipes.md](debug-recipes.md) for
|
|
||||||
forensics techniques.
|
|
||||||
|
|
@ -1,100 +0,0 @@
|
||||||
# Debug recipes: actor-system forensics
|
|
||||||
|
|
||||||
Field-tested techniques for diagnosing hangs, wedges
|
|
||||||
and cross-actor state bugs WITHOUT a debugger attached
|
|
||||||
(or when `py-spy` ain't installed).
|
|
||||||
|
|
||||||
## Wedged actor triage (no REPL)
|
|
||||||
|
|
||||||
1. find the tree:
|
|
||||||
`ps -eo pid,etime,args | grep -E 'pytest|tractor._child'`
|
|
||||||
— long-`etime` `tractor._child` procs w/ a stuck
|
|
||||||
parent = wedge.
|
|
||||||
2. kernel state:
|
|
||||||
`cat /proc/<pid>/wchan` + `status | grep -E
|
|
||||||
'State|Threads'` — `do_epoll_wait` + sleeping =
|
|
||||||
idle event loop, NOT cpu-spin.
|
|
||||||
3. **the money read** — socket queues:
|
|
||||||
`ss -tnp | grep <pid>`
|
|
||||||
- `Recv-Q > 0` on the parent-IPC conn = the actor
|
|
||||||
STOPPED CONSUMING its msg loop (runtime bug),
|
|
||||||
parent is waiting on it.
|
|
||||||
- zero external (api/ws) conns = wedged before/
|
|
||||||
without provider IO; don't blame the network.
|
|
||||||
- `CLOSE-WAIT` lingerers = unclean peer teardown.
|
|
||||||
4. cleanup: `pkill -f tractor._child` (NB: in
|
|
||||||
compound shell cmds `pkill`'s exit code poisons
|
|
||||||
`&&` chains — run it standalone).
|
|
||||||
|
|
||||||
## Hang-proof test gating
|
|
||||||
|
|
||||||
- per-suite, never combined (cross-suite session
|
|
||||||
state interacts w/ the 2nd-boot wedge):
|
|
||||||
`timeout -k 5 300 python -m pytest tests/<one>.py -q`
|
|
||||||
- rc 124/143 = hang-kill -> retry ONCE before
|
|
||||||
investigating.
|
|
||||||
- isolate a flaky test w/ a 3x loop; ~50% hit-rate
|
|
||||||
signatures match the known 2nd-boot wedge (see
|
|
||||||
gotchas.md).
|
|
||||||
|
|
||||||
## Regression vs pre-existing attribution
|
|
||||||
|
|
||||||
When a failure appears mid-refactor:
|
|
||||||
1. `git stash -u` (or checkout the file subset) and
|
|
||||||
re-run the EXACT failing case at baseline.
|
|
||||||
2. if baseline can't even run, selectively revert
|
|
||||||
ONLY the suspect layer:
|
|
||||||
`git diff <files> > /tmp/x.patch;
|
|
||||||
git checkout <files>` -> test ->
|
|
||||||
`git apply /tmp/x.patch`.
|
|
||||||
3. flake-rate compare (3x runs) beats single-shot
|
|
||||||
conclusions.
|
|
||||||
|
|
||||||
## Off-by-one / stale IPC resp detection
|
|
||||||
|
|
||||||
Mismatched query->result content in logs (resp
|
|
||||||
payload obviously for a prior request) = shared
|
|
||||||
req/resp channel w/o correlation + a cancelled
|
|
||||||
caller. Grep the ep for `move_on_after`/`fail_after`
|
|
||||||
around proxied calls. Fix = req-id (`mid`) tagging,
|
|
||||||
never "just a lock" (cancellation still orphans).
|
|
||||||
|
|
||||||
## Logging-chain audits
|
|
||||||
|
|
||||||
When records double-print or go bare (see gotchas.md):
|
|
||||||
|
|
||||||
```python
|
|
||||||
import logging
|
|
||||||
l = logging.getLogger('piker.brokers.ib.broker')
|
|
||||||
while l:
|
|
||||||
print(l.name, l.level, l.handlers, l.propagate)
|
|
||||||
l = l.parent
|
|
||||||
```
|
|
||||||
|
|
||||||
Exactly ONE stderr handler should exist in the chain,
|
|
||||||
attached by the actor's daemon fixture.
|
|
||||||
|
|
||||||
## Live actor-tree smoke (headless)
|
|
||||||
|
|
||||||
Boot against an ALT registry port so a user's running
|
|
||||||
stack is untouched; script in a REAL file (tractor
|
|
||||||
children re-exec `__main__` from path — stdin scripts
|
|
||||||
crash w/ `FileNotFoundError: .../<stdin>`):
|
|
||||||
|
|
||||||
```python
|
|
||||||
async with maybe_open_pikerd(
|
|
||||||
registry_addrs=[('127.0.0.1', 6979)],
|
|
||||||
):
|
|
||||||
async with open_feed(['xbtusdt.kraken']) as feed:
|
|
||||||
assert await check_for_service('datad.kraken')
|
|
||||||
assert not await check_for_service(
|
|
||||||
'brokerd.kraken'
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
## In-proc fail-fast unit checks
|
|
||||||
|
|
||||||
Spawn-path guards that raise BEFORE touching the
|
|
||||||
runtime can be tested w/ a bare `trio.run()` (eg
|
|
||||||
`spawn_brokerd('kucoin')` raising the datad-only
|
|
||||||
error) — no pikerd needed.
|
|
||||||
|
|
@ -1,116 +0,0 @@
|
||||||
# Known gotchas: symptom -> cause -> fix
|
|
||||||
|
|
||||||
A registry of distributed-runtime failure modes hit
|
|
||||||
(and diagnosed) in the field; check here FIRST when a
|
|
||||||
log/traceback matches.
|
|
||||||
|
|
||||||
## "Can not order ..., no qualified contract cached"
|
|
||||||
|
|
||||||
- **Symptom**: `RuntimeError` from
|
|
||||||
`ib.api.Client.submit_limit()` w/ empty
|
|
||||||
`Client._contracts` in `brokerd.ib`.
|
|
||||||
- **Cause**: per-actor cache never warmed; feed-side
|
|
||||||
qualification now lives in `datad.ib`.
|
|
||||||
- **Fix(ed)**: eager warmup at `open_trade_dialog()`
|
|
||||||
start + lazy per-order `get_mkt_info()` +
|
|
||||||
`cache_contract()` (writes BOTH `mkt.bs_fqme` and
|
|
||||||
`mkt.fqme` keys; different consumers read each!).
|
|
||||||
|
|
||||||
## Search returns results for the WRONG pattern
|
|
||||||
|
|
||||||
- **Symptom**: fqme search for 'gld' returns nvda
|
|
||||||
results; next query returns the prior query's set.
|
|
||||||
- **Cause**: `MethodProxy` channel off-by-one — a
|
|
||||||
caller cancelled (search `move_on_after` timeout)
|
|
||||||
after sending its request orphans the response;
|
|
||||||
every later caller consumes the previous resp.
|
|
||||||
- **Fix(ed)**: `mid` req-id correlation in
|
|
||||||
`_run_method()` + relay; stale resps are dropped w/
|
|
||||||
a "Dropping stale method-resp" warning. If that
|
|
||||||
warning spams, some caller is being cancelled
|
|
||||||
mid-call habitually — find + fix its timeout.
|
|
||||||
|
|
||||||
## One bad request crashes a whole dialog/actor
|
|
||||||
|
|
||||||
- **Symptom**: `TrioTaskExited` storm + nursery
|
|
||||||
teardown after a single method error (eg ambiguous
|
|
||||||
contract `AttributeError`).
|
|
||||||
- **Cause**: exception escaped the aio-side relay
|
|
||||||
loop (`open_aio_client_method_relay()`) killing
|
|
||||||
channel + proxy nursery; caller-side `try/except`
|
|
||||||
CANNOT catch it.
|
|
||||||
- **Fix(ed)**: relay catches `Exception` -> ships
|
|
||||||
`{'exception': err, 'mid': ...}` resp; order
|
|
||||||
handler converts to EMS `BrokerdError` msgs.
|
|
||||||
|
|
||||||
## Ambiguous ib contracts -> `NoneType` attr errors
|
|
||||||
|
|
||||||
- **Symptom**: `'NoneType' object has no attribute
|
|
||||||
'primaryExchange'` in `find_contracts()`.
|
|
||||||
- **Cause**: `qualifyContractsAsync()` returns `None`
|
|
||||||
entries for ambiguous (eg venue-less stonk fqme
|
|
||||||
matching multiple listings: 'gld' -> ARCA/USD +
|
|
||||||
VENTURE/CAD).
|
|
||||||
- **Fix(ed)**: filter `None`s + raise descriptive
|
|
||||||
`ValueError` ("use 'gld.arca.ib'").
|
|
||||||
|
|
||||||
## Double-printed log records (same task id, 2x)
|
|
||||||
|
|
||||||
- **Symptom**: every record from some subsys printed
|
|
||||||
twice w/ identical task ids.
|
|
||||||
- **Cause**: stderr handlers attached at TWO levels
|
|
||||||
of one logger-propagation chain (eg daemon fixture
|
|
||||||
on `piker.brokers.ib` + an ep calling
|
|
||||||
`get_console_log(name=__name__)` on the child).
|
|
||||||
tractor's handler-dedup only checks the SAME
|
|
||||||
logger, not ancestors.
|
|
||||||
- **Rule**: console handlers are attached ONCE per
|
|
||||||
actor in the `_setup_persistent_*()` fixture; eps
|
|
||||||
needing a different level use `log.setLevel()`
|
|
||||||
ONLY, never `get_console_log()`.
|
|
||||||
|
|
||||||
## Bare/non-colorized log lines
|
|
||||||
|
|
||||||
- **Symptom**: records w/ no timestamp/actor prefix.
|
|
||||||
- **Cause**: NO handler anywhere in the emitting
|
|
||||||
logger's chain -> stdlib `logging.lastResort`. Post
|
|
||||||
actor-splits, a daemon fixture may only cover its
|
|
||||||
own subsys subtree (eg datad's `piker.data.*` but
|
|
||||||
not the backend's `piker.brokers.<broker>.*`).
|
|
||||||
- **Fix(ed)**: `_setup_persistent_datad()` enables
|
|
||||||
BOTH `piker.data.<broker>` and
|
|
||||||
`piker.brokers.<broker>` subtrees.
|
|
||||||
|
|
||||||
## 2nd in-proc runtime boot wedges (~50%)
|
|
||||||
|
|
||||||
- **Symptom**: test hangs when one test proc boots a
|
|
||||||
2nd `pikerd` (eg `test_multi_fill_positions`'s
|
|
||||||
persistence re-check); a zombie `*.{broker}` child
|
|
||||||
lingers w/ unread bytes in its parent-IPC Recv-Q.
|
|
||||||
- **Cause**: pre-existing `tractor`-main runtime
|
|
||||||
teardown bug (confirmed independent of piker-layer
|
|
||||||
changes via revert-testing 2026-06).
|
|
||||||
- **Mitigation**: run suites per-file wrapped in
|
|
||||||
`timeout -k 5 300 ...`; retry once on rc 124/143.
|
|
||||||
Do NOT chase as a regression of unrelated changes.
|
|
||||||
|
|
||||||
## ib client-id collisions post-split
|
|
||||||
|
|
||||||
- **Symptom**: 2nd ib daemon burns the full
|
|
||||||
conn-timeout retry cycle connecting to gw/tws.
|
|
||||||
- **Cause**: `datad.ib` + `brokerd.ib` both default
|
|
||||||
`client_id=6116` w/ linear `+i` retries.
|
|
||||||
- **Fix(ed)**: role-based offsets in
|
|
||||||
`load_aio_clients()`: datad +16, ad-hoc (test/cli)
|
|
||||||
actors +32.
|
|
||||||
|
|
||||||
## `async_lifo_cache` skipped side-effects
|
|
||||||
|
|
||||||
- **Symptom**: a fn's cache-write side effect
|
|
||||||
(eg `get_mkt_info()` -> `_contracts`) missing for
|
|
||||||
a 2nd client/proxy.
|
|
||||||
- **Cause**: cache keys on POSITIONAL args only; a
|
|
||||||
hit skips the body entirely.
|
|
||||||
- **Rule**: never rely on cached-fn side effects;
|
|
||||||
perform required writes explicitly at the call
|
|
||||||
site (eg `cache_contract()` after `get_mkt_info`).
|
|
||||||
|
|
@ -5,11 +5,7 @@ description: >
|
||||||
across distributed actor systems. Apply when
|
across distributed actor systems. Apply when
|
||||||
adding profiling, debugging perf regressions, or
|
adding profiling, debugging perf regressions, or
|
||||||
optimizing hot paths in piker code.
|
optimizing hot paths in piker code.
|
||||||
compatibility: >
|
user-invocable: false
|
||||||
Requires a piker checkout and Python profiling work.
|
|
||||||
metadata:
|
|
||||||
author: goodboy
|
|
||||||
version: "1.0"
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Piker Profiling Subsystem
|
# Piker Profiling Subsystem
|
||||||
|
|
|
||||||
|
|
@ -5,11 +5,7 @@ description: >
|
||||||
ethos. Apply when communicating with piker devs,
|
ethos. Apply when communicating with piker devs,
|
||||||
writing commit messages, code review comments, or
|
writing commit messages, code review comments, or
|
||||||
any collaborative interaction.
|
any collaborative interaction.
|
||||||
compatibility: >
|
user-invocable: false
|
||||||
Harness-independent communication guidance.
|
|
||||||
metadata:
|
|
||||||
author: goodboy
|
|
||||||
version: "1.0"
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Piker Slang & Communication Style
|
# Piker Slang & Communication Style
|
||||||
|
|
|
||||||
|
|
@ -5,12 +5,7 @@ description: >
|
||||||
for piker's UI. Apply when optimizing graphics
|
for piker's UI. Apply when optimizing graphics
|
||||||
performance, adding new chart annotations, or
|
performance, adding new chart annotations, or
|
||||||
working with `QGraphicsItem` subclasses.
|
working with `QGraphicsItem` subclasses.
|
||||||
compatibility: >
|
user-invocable: false
|
||||||
Requires a piker checkout with PyQtGraph and Qt source
|
|
||||||
available for inspection.
|
|
||||||
metadata:
|
|
||||||
author: goodboy
|
|
||||||
version: "1.0"
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# PyQtGraph Rendering Optimization
|
# PyQtGraph Rendering Optimization
|
||||||
|
|
|
||||||
|
|
@ -1,285 +0,0 @@
|
||||||
# Test Harness Reference: piker
|
|
||||||
|
|
||||||
This repository-local file supplements the canonical `/run-tests` skill.
|
|
||||||
Keep shared execution, worktree, failure-inspection, and cleanup policy in the
|
|
||||||
canonical `SKILL.md`; keep Piker commands, paths, fixtures, and known outcomes
|
|
||||||
here.
|
|
||||||
|
|
||||||
## Project And Environment
|
|
||||||
|
|
||||||
- Project/import: `piker`
|
|
||||||
- Test root: `tests/`
|
|
||||||
- Supported Python: `>=3.12,<3.14`
|
|
||||||
- Preferred complete environment: worktree-local `py313` inside the current
|
|
||||||
`nix develop` shell
|
|
||||||
- The flake shell pins CPython 3.13 and sets
|
|
||||||
`UV_PROJECT_ENVIRONMENT=py313`.
|
|
||||||
- Verify the interpreter, package resolution, and dependency import before
|
|
||||||
running tests. A bare `py313` may lack the Qt binding supplied by Nix.
|
|
||||||
|
|
||||||
Use an already-provisioned `py313` only when `import piker` succeeds:
|
|
||||||
|
|
||||||
```text
|
|
||||||
py313/bin/python -m pytest -p no:xonsh
|
|
||||||
```
|
|
||||||
|
|
||||||
If direct environment paths are unavailable, an existing uv environment can
|
|
||||||
be used without changing it, subject to the same import check:
|
|
||||||
|
|
||||||
```text
|
|
||||||
UV_PROJECT_ENVIRONMENT=py313 uv run --frozen --no-sync python -m pytest -p no:xonsh
|
|
||||||
```
|
|
||||||
|
|
||||||
Ask before running provisioning commands such as:
|
|
||||||
|
|
||||||
```text
|
|
||||||
UV_PROJECT_ENVIRONMENT=py313 uv sync --dev --all-extras --no-group lint
|
|
||||||
nix develop
|
|
||||||
nix-shell default.nix
|
|
||||||
```
|
|
||||||
|
|
||||||
`nix develop` is the current Wayland/Qt 6 shell. `default.nix` is the current
|
|
||||||
X11 shell. Do not use `develop.nix` for current testing; it retains the old
|
|
||||||
Python 3.11, Poetry, and Qt 5 stack.
|
|
||||||
|
|
||||||
The current root-checkout `py313` resolves `piker` locally but fails
|
|
||||||
`import piker` outside Nix because PyQtGraph cannot import PyQt or PySide. Do
|
|
||||||
not treat that environment as test-ready and do not enter `nix develop`
|
|
||||||
without approval: its shell hook may recreate and sync `py313`.
|
|
||||||
|
|
||||||
Plain `uv sync` does not include the testing group. The `dbs` dependency group
|
|
||||||
is also absent from normal dev-shell provisioning.
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
Base command in the preferred environment:
|
|
||||||
|
|
||||||
```text
|
|
||||||
py313/bin/python -m pytest -p no:xonsh
|
|
||||||
```
|
|
||||||
|
|
||||||
The explicit `-p no:xonsh` is required. The tracked comments-only
|
|
||||||
`pytest.ini` takes precedence over `pyproject.toml`, so the intended
|
|
||||||
`addopts = "-p no:xonsh"` and `testpaths = ["tests"]` are inactive.
|
|
||||||
Always pass a test path or node ID explicitly.
|
|
||||||
|
|
||||||
Package-resolution check that does not import Piker's dependencies:
|
|
||||||
|
|
||||||
```text
|
|
||||||
py313/bin/python -c 'import importlib.util, pathlib, sys; root = pathlib.Path.cwd().resolve(); spec = importlib.util.find_spec("piker"); mod = pathlib.Path(spec.origin).resolve(); print(sys.executable); print(mod); assert mod.is_relative_to(root)'
|
|
||||||
```
|
|
||||||
|
|
||||||
Dependency import check, required before collection or execution:
|
|
||||||
|
|
||||||
```text
|
|
||||||
py313/bin/python -c 'import pathlib, piker, sys; root = pathlib.Path.cwd().resolve(); mod = pathlib.Path(piker.__file__).resolve(); print(sys.executable); print(mod); assert mod.is_relative_to(root)'
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe core collection check:
|
|
||||||
|
|
||||||
```text
|
|
||||||
py313/bin/python -m pytest -p no:xonsh -q --collect-only tests/test_watchlists.py tests/test_accounting.py tests/test_services.py tests/test_ems.py tests/test_feeds.py tests/test_cli.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Default first-pass flags are `-q -x --tb=short --no-header` unless the user
|
|
||||||
requests otherwise. For actor-heavy tests, use one file or node per process
|
|
||||||
with an outer timeout:
|
|
||||||
|
|
||||||
```text
|
|
||||||
timeout -k 5 300 py313/bin/python -m pytest -p no:xonsh -q <one-file-or-node>
|
|
||||||
```
|
|
||||||
|
|
||||||
If an actor-heavy command exits `124` or `143`, retry that exact command once
|
|
||||||
and report both attempts. Never convert a retry pass into an unconditional
|
|
||||||
pass, and do not retry assertion, import, collection, or configuration
|
|
||||||
failures.
|
|
||||||
|
|
||||||
## Scope And Path Resolution
|
|
||||||
|
|
||||||
Resolve bare test filenames beneath `tests/`. Preserve complete node IDs and
|
|
||||||
apply `-k` only within explicitly selected paths. There is no marker-based
|
|
||||||
offline/full-suite split, so never use `pytest tests` as a deterministic
|
|
||||||
default.
|
|
||||||
|
|
||||||
Deterministic or local first-pass targets:
|
|
||||||
|
|
||||||
- `tests/test_watchlists.py`
|
|
||||||
- `tests/test_storage_audit.py`
|
|
||||||
- `tests/test_backfill_audit_snippet.py`
|
|
||||||
- `tests/test_ib_history.py`
|
|
||||||
- `tests/test_history_backfill.py`
|
|
||||||
- `tests/test_ldshm.py`
|
|
||||||
- `tests/test_accounting.py::test_account_file_default_empty`
|
|
||||||
- `tests/test_services.py::test_runtime_boot`
|
|
||||||
- `tests/test_services.py::test_datad_spawn`
|
|
||||||
- `tests/test_ems.py::test_ems_err_on_bad_broker`
|
|
||||||
|
|
||||||
Require explicit authorization before running:
|
|
||||||
|
|
||||||
- `tests/test_feeds.py` - live Binance/Kraken feeds;
|
|
||||||
- `tests/test_services.py::test_ensure_datafeed_actors` - live Kraken feed;
|
|
||||||
- `tests/test_services.py::test_ensure_ems_in_paper_actors` - paper EMS with
|
|
||||||
live Kraken symbology/feed access;
|
|
||||||
- `tests/test_ems.py::test_multi_fill_positions` - live backend setup and
|
|
||||||
persisted state;
|
|
||||||
- `tests/test_accounting.py::test_paper_ledger_position_calcs` - tracked
|
|
||||||
fixtures plus possible live symcache generation;
|
|
||||||
- `tests/test_accounting.py::test_ib_account_with_duplicated_mktids` - active
|
|
||||||
broker/account configuration and state writes;
|
|
||||||
- `tests/test_dpi_font.py` - Qt/UI import and user-config side effects;
|
|
||||||
- `tests/test_docker_services.py` - optional dependencies and containers;
|
|
||||||
- `tests/test_questrade.py` - obsolete credentialed imports.
|
|
||||||
|
|
||||||
`tests/test_cli.py` is currently hard-skipped. It is not an active CLI
|
|
||||||
regression gate.
|
|
||||||
|
|
||||||
Never execute `piker store anal` or `piker store ldshm` as tests. They are
|
|
||||||
mutating or interactive operational commands.
|
|
||||||
|
|
||||||
## Project-Specific Flags And Backend Matrix
|
|
||||||
|
|
||||||
| Flag | Purpose |
|
|
||||||
|---|---|
|
|
||||||
| `--ll LEVEL` | Piker log level |
|
|
||||||
| `--confdir PATH` | Override `piker.config._config_dir` |
|
|
||||||
| `--spawn-backend trio|mp_spawn|mp_forkserver` | Tractor process backend |
|
|
||||||
| `--tpt-proto PROTO` | Tractor transport; one protocol per session |
|
|
||||||
| `--tpdb` / `--debug-mode` | Tractor plugin debug mode |
|
|
||||||
| `--pdb` | Standard pytest debugger |
|
|
||||||
| `-s` | No capture; required with `--pdb` and `open_test_pikerd` |
|
|
||||||
|
|
||||||
Do not invent `network`, `offline`, `docker`, `gui`, or `broker` markers; none
|
|
||||||
currently exists. `CI=1` is not an offline selector: feed tests still leave a
|
|
||||||
live Kraken case enabled. Current Piker tests exercise TCP; do not assume the
|
|
||||||
whole suite supports UDS merely because the Tractor plugin exposes it.
|
|
||||||
|
|
||||||
## Fixture Invariants
|
|
||||||
|
|
||||||
- The session `confdir` fixture does nothing unless `--confdir` is passed.
|
|
||||||
Its claimed `tests/data` fallback is not implemented and that directory is
|
|
||||||
absent.
|
|
||||||
- Function-scoped `tmpconfdir` changes process-global config state and does
|
|
||||||
not restore the previous path. Use separate pytest processes when
|
|
||||||
diagnosing state leakage.
|
|
||||||
- `open_test_pikerd` passes the temporary config path to child actors through
|
|
||||||
`tractor_runtime_overrides`.
|
|
||||||
- `tests/_inputs/trades_binance_paper.toml` and
|
|
||||||
`tests/_inputs/account.binance.paper.toml` are used in place. Accounting
|
|
||||||
contexts can write them on exit. Inspect `git diff -- tests/_inputs` after
|
|
||||||
any selected accounting case.
|
|
||||||
- Importing `tests/test_dpi_font.py` constructs module-global font objects
|
|
||||||
before fixtures can isolate config. If explicitly requested, isolate
|
|
||||||
`XDG_CONFIG_HOME` before Python starts and use the proper Qt/Nix shell.
|
|
||||||
- Piker and the installed Tractor pytest plugin do not provide a
|
|
||||||
repository-local process or socket reaper. Never apply historical broad
|
|
||||||
`pkill -f tractor._child` guidance automatically.
|
|
||||||
- The function-scoped autouse `shm_leak_tracker` fixture wraps Tractor's
|
|
||||||
current-process `SharedMemory` factory. It tracks only successful
|
|
||||||
`create=True` calls, restores the pre-test `_known_tokens` cache, and
|
|
||||||
unlinks exact surviving names before failing the leaking test. It never
|
|
||||||
scans `/dev/shm` or unlinks attachments created by another process.
|
|
||||||
|
|
||||||
## Test Layout
|
|
||||||
|
|
||||||
```text
|
|
||||||
tests/
|
|
||||||
conftest.py Piker options, config fixtures, Tractor plugin
|
|
||||||
_inputs/ tracked ledger/account fixtures
|
|
||||||
test_accounting.py config, ledgers, accounts, and position math
|
|
||||||
test_cli.py legacy CLI suite; hard-skipped
|
|
||||||
test_docker_services.py container integrations; optional deps, skipped
|
|
||||||
test_dpi_font.py Qt DPI/font behavior
|
|
||||||
test_ems.py actor, EMS, and paper-position behavior
|
|
||||||
test_feeds.py live Binance/Kraken feeds and shared memory
|
|
||||||
test_ib_history.py deterministic IB history request formatting
|
|
||||||
test_history_backfill.py deterministic history/SHM orchestration
|
|
||||||
test_ldshm.py ldshm unpublished-slot guard
|
|
||||||
test_questrade.py obsolete credentialed tests; skipped
|
|
||||||
test_services.py pikerd/datad/feed/EMS actor lifecycle
|
|
||||||
test_storage_audit.py read-only NativeDB audit and JSON CLI
|
|
||||||
test_backfill_audit_snippet.py
|
|
||||||
disposable xonsh qualification helpers
|
|
||||||
test_watchlists.py deterministic watchlist JSON operations
|
|
||||||
```
|
|
||||||
|
|
||||||
## Change-To-Test Mapping
|
|
||||||
|
|
||||||
| Changed area | Run first | Caveat |
|
|
||||||
|---|---|---|
|
|
||||||
| `piker/watchlists/` | `tests/test_watchlists.py` | CLI suite is skipped |
|
|
||||||
| `piker/storage/_audit.py`, `piker/storage/cli.py` | `tests/test_storage_audit.py` | direct Typer app, no actor |
|
|
||||||
| `snippets/nativedb_backfill_audit.xsh` | `tests/test_backfill_audit_snippet.py` | disposable paths only |
|
|
||||||
| `piker/brokers/ib/api.py`, `feed.py` history | `tests/test_ib_history.py` | fake client, no network |
|
|
||||||
| `piker/tsp/_history.py` | `tests/test_history_backfill.py` | fake provider/storage/SHM |
|
|
||||||
| `piker/storage/cli.py` ldshm null-slot guard | `tests/test_ldshm.py` | synthetic timestamps, no SHM mutation |
|
|
||||||
| `piker/config.py` | `test_account_file_default_empty` | root-network test has a known mismatch |
|
|
||||||
| `piker/accounting/` | targeted accounting node | some cases use live/configured state |
|
|
||||||
| `piker/ui/_style.py`, `piker/ui/qt.py` | `tests/test_dpi_font.py` | GUI/config-isolated opt-in |
|
|
||||||
| `piker/service/_actor_runtime.py`, `_registry.py`, `_mngr.py` | `test_runtime_boot` | then `test_datad_spawn` |
|
|
||||||
| `piker/service/`, `piker/data/_daemon.py` | `test_datad_spawn` | feed lifecycle cases are live |
|
|
||||||
| `piker/data/feed.py`, `flows.py`, `_sharedmem.py`, `_sampling.py` | collect first | feed execution needs live permission |
|
|
||||||
| `piker/clearing/` | `test_ems_err_on_bad_broker` | multi-fill case is live/persisted |
|
|
||||||
| `piker/brokers/binance/`, `kraken/` | selected feed/accounting node | live network |
|
|
||||||
| `piker/brokers/ib/` | duplicated-market-ID node | controlled account config required |
|
|
||||||
| Docker/service adapters | `tests/test_docker_services.py` | optional deps and containers |
|
|
||||||
| project, lock, or Nix files | import check and safe collection | full collection is not safe by default |
|
|
||||||
|
|
||||||
Prefer deterministic filesystem/config tests, then local actor-runtime nodes,
|
|
||||||
then explicitly approved live broker, GUI, or container coverage.
|
|
||||||
|
|
||||||
## Quick Checks
|
|
||||||
|
|
||||||
```text
|
|
||||||
py313/bin/python -c 'import importlib.util, pathlib, sys; root = pathlib.Path.cwd().resolve(); spec = importlib.util.find_spec("piker"); mod = pathlib.Path(spec.origin).resolve(); print(sys.executable); print(mod); assert mod.is_relative_to(root)'
|
|
||||||
py313/bin/python -c 'import pathlib, piker, sys; root = pathlib.Path.cwd().resolve(); mod = pathlib.Path(piker.__file__).resolve(); print(sys.executable); print(mod); assert mod.is_relative_to(root)'
|
|
||||||
py313/bin/python -m pytest -p no:xonsh -q tests/test_watchlists.py
|
|
||||||
py313/bin/python -m pytest -p no:xonsh -q tests/test_accounting.py::test_account_file_default_empty
|
|
||||||
timeout -k 5 300 py313/bin/python -m pytest -p no:xonsh -q tests/test_services.py::test_runtime_boot
|
|
||||||
timeout -k 5 300 py313/bin/python -m pytest -p no:xonsh -q tests/test_services.py::test_datad_spawn
|
|
||||||
timeout -k 5 300 py313/bin/python -m pytest -p no:xonsh -q tests/test_ems.py::test_ems_err_on_bad_broker
|
|
||||||
```
|
|
||||||
|
|
||||||
## Known Outcomes
|
|
||||||
|
|
||||||
- The current root-checkout `py313` fails `import piker` outside the Nix shell
|
|
||||||
with `ImportError: PyQtGraph requires one of PyQt5, PyQt6, PySide2 or
|
|
||||||
PySide6`. This is an incomplete environment, not an application regression.
|
|
||||||
- `tests/test_accounting.py::test_root_conf_networking_section` currently
|
|
||||||
expects `network.tsdb`, which is absent from the tracked config template.
|
|
||||||
Match the current `KeyError: 'tsdb'` before classifying it as the known
|
|
||||||
repository mismatch.
|
|
||||||
- `tests/test_docker_services.py` is marked skipped but imports
|
|
||||||
`elasticsearch` first. Without the `dbs` group it fails collection rather
|
|
||||||
than skipping.
|
|
||||||
- `tests/test_questrade.py` is marked skipped but imports undeclared `asks`
|
|
||||||
through the legacy broker module before marks apply.
|
|
||||||
- `test_open_orders_reloaded` and `test_dark_order_clearing` in
|
|
||||||
`tests/test_ems.py` contain only ellipsis bodies. A pass does not verify the
|
|
||||||
named behavior.
|
|
||||||
- Treat `.pytest_cache` feed IDs using old FQME forms as stale historical
|
|
||||||
cache, not current known failures.
|
|
||||||
|
|
||||||
Do not classify every `TooSlowError`, timeout, or child survivor as the known
|
|
||||||
Tractor teardown wedge. Match the selected node and the documented
|
|
||||||
second-runtime/lingering-child signature.
|
|
||||||
|
|
||||||
## Tractor Runtime Notes
|
|
||||||
|
|
||||||
The suite loads `tractor._testing.pytest`. Defaults are the `trio` spawn
|
|
||||||
backend and TCP transport. Registry addresses are normally session-unique,
|
|
||||||
except `tests/test_services.py::test_runtime_boot`, which binds
|
|
||||||
`127.0.0.1:6666`. Check that fixed port only for that node; ordinary tests do
|
|
||||||
not require Piker's production `127.0.0.1:6116` registry address.
|
|
||||||
|
|
||||||
Current repository concurrency notes document an intermittent second
|
|
||||||
in-process `pikerd` boot wedge with a lingering broker child and unread parent
|
|
||||||
IPC bytes. Use the outer timeout for actor-heavy nodes. Retry an exact command
|
|
||||||
once only after status `124` or `143`, and report both attempts.
|
|
||||||
|
|
||||||
Ordinary actor tests support normal capture. Diagnose a capture-dependent hang
|
|
||||||
by retrying only the exact node with `-s`. Standard `--pdb` with
|
|
||||||
`open_test_pikerd` requires `-s`; Tractor's `--tpdb` is a separate option.
|
|
||||||
|
|
||||||
After abnormal exit, inspect only descendants, sockets, and shared-memory
|
|
||||||
objects attributable to the exact pytest session. Ask before signaling or
|
|
||||||
unlinking anything.
|
|
||||||
|
|
@ -6,12 +6,7 @@ description: >
|
||||||
with OHLCV arrays, timestamp lookups, gap
|
with OHLCV arrays, timestamp lookups, gap
|
||||||
detection, or any array/dataframe operations in
|
detection, or any array/dataframe operations in
|
||||||
piker.
|
piker.
|
||||||
compatibility: >
|
user-invocable: false
|
||||||
Requires a piker checkout with NumPy and optionally
|
|
||||||
Polars.
|
|
||||||
metadata:
|
|
||||||
author: goodboy
|
|
||||||
version: "1.0"
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Timeseries Optimization: NumPy & Polars
|
# Timeseries Optimization: NumPy & Polars
|
||||||
|
|
@ -66,14 +61,11 @@ ts_array = np.array(timestamps)
|
||||||
# binary search for all timestamps at once
|
# binary search for all timestamps at once
|
||||||
indices = np.searchsorted(time_arr, ts_array)
|
indices = np.searchsorted(time_arr, ts_array)
|
||||||
|
|
||||||
# bounds check before indexing: searchsorted may return
|
# bounds check and exact match verification
|
||||||
# len(time_arr) for values above the final timestamp
|
valid_mask = (
|
||||||
in_bounds = indices < len(time_arr)
|
(indices < len(array))
|
||||||
valid_mask = np.zeros(indices.shape, dtype=bool)
|
&
|
||||||
valid_mask[in_bounds] = (
|
(time_arr[indices] == ts_array)
|
||||||
time_arr[indices[in_bounds]]
|
|
||||||
==
|
|
||||||
ts_array[in_bounds]
|
|
||||||
)
|
)
|
||||||
|
|
||||||
valid_indices = indices[valid_mask]
|
valid_indices = indices[valid_mask]
|
||||||
|
|
|
||||||
|
|
@ -110,12 +110,6 @@ ENV/
|
||||||
.claude/*_commit_*.md
|
.claude/*_commit_*.md
|
||||||
.claude/*_commit*.toml
|
.claude/*_commit*.toml
|
||||||
|
|
||||||
# ai.skillz/commit-msg
|
|
||||||
.claude/skills/commit-msg/msgs/
|
|
||||||
.claude/skills/commit-msg/conf.toml
|
|
||||||
.claude/git_commit_msg_LATEST.md
|
|
||||||
.opencode/skills/commit-msg/SKILL.md
|
|
||||||
|
|
||||||
# nix develop --profile .nixdev
|
# nix develop --profile .nixdev
|
||||||
.nixdev*
|
.nixdev*
|
||||||
|
|
||||||
|
|
@ -136,172 +130,3 @@ gitea/
|
||||||
|
|
||||||
# LLM conversations that should remain private
|
# LLM conversations that should remain private
|
||||||
docs/conversations/
|
docs/conversations/
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:run-tests
|
|
||||||
/.claude/skills/run-tests/SKILL.md
|
|
||||||
# END ai.skillz: direct:symlink:claude:run-tests
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:run-tests
|
|
||||||
/.opencode/skills/run-tests/SKILL.md
|
|
||||||
# END ai.skillz: direct:symlink:opencode:run-tests
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:command:run-tests
|
|
||||||
/.opencode/commands/run-tests.md
|
|
||||||
# END ai.skillz: direct:symlink:opencode:command:run-tests
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:close-wkt
|
|
||||||
/.claude/skills/close-wkt
|
|
||||||
# END ai.skillz: direct:symlink:claude:close-wkt
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:close-wkt
|
|
||||||
/.opencode/skills/close-wkt
|
|
||||||
# END ai.skillz: direct:symlink:opencode:close-wkt
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: runtime:code-review-changes
|
|
||||||
.claude/review_context.md
|
|
||||||
.claude/review_regression.md
|
|
||||||
# END ai.skillz: runtime:code-review-changes
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:code-review-changes
|
|
||||||
/.claude/skills/code-review-changes
|
|
||||||
# END ai.skillz: direct:symlink:claude:code-review-changes
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:code-review-changes
|
|
||||||
/.opencode/skills/code-review-changes
|
|
||||||
# END ai.skillz: direct:symlink:opencode:code-review-changes
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: runtime:commit-msg
|
|
||||||
.claude/skills/commit-msg/msgs/
|
|
||||||
.claude/skills/commit-msg/conf.toml
|
|
||||||
.claude/git_commit_msg_LATEST.md
|
|
||||||
# END ai.skillz: runtime:commit-msg
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:commit-msg
|
|
||||||
/.claude/skills/commit-msg/SKILL.md
|
|
||||||
# END ai.skillz: direct:symlink:claude:commit-msg
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:commit-msg
|
|
||||||
/.opencode/skills/commit-msg/SKILL.md
|
|
||||||
# END ai.skillz: direct:symlink:opencode:commit-msg
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:dep-supersede-scan
|
|
||||||
/.claude/skills/dep-supersede-scan
|
|
||||||
# END ai.skillz: direct:symlink:claude:dep-supersede-scan
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:dep-supersede-scan
|
|
||||||
/.opencode/skills/dep-supersede-scan
|
|
||||||
# END ai.skillz: direct:symlink:opencode:dep-supersede-scan
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:gish
|
|
||||||
/.claude/skills/gish
|
|
||||||
# END ai.skillz: direct:symlink:claude:gish
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:gish
|
|
||||||
/.opencode/skills/gish
|
|
||||||
# END ai.skillz: direct:symlink:opencode:gish
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:inter-skill-review
|
|
||||||
/.claude/skills/inter-skill-review
|
|
||||||
# END ai.skillz: direct:symlink:claude:inter-skill-review
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:inter-skill-review
|
|
||||||
/.opencode/skills/inter-skill-review
|
|
||||||
# END ai.skillz: direct:symlink:opencode:inter-skill-review
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: runtime:open-wkt
|
|
||||||
.claude/wkts/
|
|
||||||
claude_wkts
|
|
||||||
# END ai.skillz: runtime:open-wkt
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:open-wkt
|
|
||||||
/.claude/skills/open-wkt
|
|
||||||
# END ai.skillz: direct:symlink:claude:open-wkt
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:open-wkt
|
|
||||||
/.opencode/skills/open-wkt
|
|
||||||
# END ai.skillz: direct:symlink:opencode:open-wkt
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:plan-io
|
|
||||||
/.claude/skills/plan-io
|
|
||||||
# END ai.skillz: direct:symlink:claude:plan-io
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:plan-io
|
|
||||||
/.opencode/skills/plan-io
|
|
||||||
# END ai.skillz: direct:symlink:opencode:plan-io
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: runtime:pr-msg
|
|
||||||
.claude/skills/pr-msg/msgs/
|
|
||||||
.claude/skills/pr-msg/pr_msg_LATEST.md
|
|
||||||
# END ai.skillz: runtime:pr-msg
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:pr-msg
|
|
||||||
/.claude/skills/pr-msg/SKILL.md
|
|
||||||
/.claude/skills/pr-msg/references
|
|
||||||
/.claude/skills/pr-msg/scripts
|
|
||||||
# END ai.skillz: direct:symlink:claude:pr-msg
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:pr-msg
|
|
||||||
/.opencode/skills/pr-msg/SKILL.md
|
|
||||||
/.opencode/skills/pr-msg/references
|
|
||||||
/.opencode/skills/pr-msg/scripts
|
|
||||||
# END ai.skillz: direct:symlink:opencode:pr-msg
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:prompt-io
|
|
||||||
/.claude/skills/prompt-io
|
|
||||||
# END ai.skillz: direct:symlink:claude:prompt-io
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:prompt-io
|
|
||||||
/.opencode/skills/prompt-io
|
|
||||||
# END ai.skillz: direct:symlink:opencode:prompt-io
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:py-codestyle
|
|
||||||
/.claude/skills/py-codestyle
|
|
||||||
# END ai.skillz: direct:symlink:claude:py-codestyle
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:py-codestyle
|
|
||||||
/.opencode/skills/py-codestyle
|
|
||||||
# END ai.skillz: direct:symlink:opencode:py-codestyle
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:resolve-conflicts
|
|
||||||
/.claude/skills/resolve-conflicts
|
|
||||||
# END ai.skillz: direct:symlink:claude:resolve-conflicts
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:resolve-conflicts
|
|
||||||
/.opencode/skills/resolve-conflicts
|
|
||||||
# END ai.skillz: direct:symlink:opencode:resolve-conflicts
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: runtime:taken-export
|
|
||||||
.ai/taken/exports/
|
|
||||||
# END ai.skillz: runtime:taken-export
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:taken-export
|
|
||||||
/.claude/skills/taken-export
|
|
||||||
# END ai.skillz: direct:symlink:claude:taken-export
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:taken-export
|
|
||||||
/.opencode/skills/taken-export
|
|
||||||
# END ai.skillz: direct:symlink:opencode:taken-export
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:yt-url-lookup
|
|
||||||
/.claude/skills/yt-url-lookup
|
|
||||||
# END ai.skillz: direct:symlink:claude:yt-url-lookup
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:yt-url-lookup
|
|
||||||
/.opencode/skills/yt-url-lookup
|
|
||||||
# END ai.skillz: direct:symlink:opencode:yt-url-lookup
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:command:branch-in-new-terminal
|
|
||||||
/.claude/commands/branch-in-new-terminal.md
|
|
||||||
# END ai.skillz: direct:symlink:claude:command:branch-in-new-terminal
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: runtime:branch-in-new-terminal
|
|
||||||
.claude/.current_session
|
|
||||||
# END ai.skillz: runtime:branch-in-new-terminal
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:command:commit-msg
|
|
||||||
/.opencode/commands/commit-msg.md
|
|
||||||
# END ai.skillz: direct:symlink:opencode:command:commit-msg
|
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:opencode:command:taken-export
|
|
||||||
/.opencode/commands/taken-export.md
|
|
||||||
# END ai.skillz: direct:symlink:opencode:command:taken-export
|
|
||||||
|
|
|
||||||
19
ai/README.md
19
ai/README.md
|
|
@ -17,18 +17,6 @@ track new integration ideas and proposals in
|
||||||
| Tool | Directory | Status |
|
| Tool | Directory | Status |
|
||||||
|------|-----------|--------|
|
|------|-----------|--------|
|
||||||
| [Claude Code](https://github.com/anthropics/claude-code) | [`claude-code/`](claude-code/) | active |
|
| [Claude Code](https://github.com/anthropics/claude-code) | [`claude-code/`](claude-code/) | active |
|
||||||
| [OpenCode](https://opencode.ai/) | [`opencode/`](opencode/) | active |
|
|
||||||
|
|
||||||
## Shared Skills
|
|
||||||
|
|
||||||
Repo-specific skills use the portable Agent Skills
|
|
||||||
frontmatter subset and live under `.claude/skills/` as a
|
|
||||||
single source. Claude Code discovers that directory
|
|
||||||
natively, and OpenCode discovers the same project skills
|
|
||||||
without copies or generated wrappers.
|
|
||||||
|
|
||||||
Harness-specific command files should only delegate to a
|
|
||||||
shared skill. They must not duplicate the skill body.
|
|
||||||
|
|
||||||
## Adding a New Integration
|
## Adding a New Integration
|
||||||
|
|
||||||
|
|
@ -44,7 +32,7 @@ ai/
|
||||||
├── README.md # <- you are here
|
├── README.md # <- you are here
|
||||||
├── claude-code/
|
├── claude-code/
|
||||||
│ └── README.md
|
│ └── README.md
|
||||||
├── opencode/
|
├── opencode/ # future
|
||||||
│ └── README.md
|
│ └── README.md
|
||||||
└── <your-tool>/
|
└── <your-tool>/
|
||||||
└── README.md
|
└── README.md
|
||||||
|
|
@ -57,7 +45,6 @@ ai/
|
||||||
- Each integration doc should describe **what**
|
- Each integration doc should describe **what**
|
||||||
the skill does, **how** to invoke it, and any
|
the skill does, **how** to invoke it, and any
|
||||||
**output** artifacts it produces
|
**output** artifacts it produces
|
||||||
- Keep docs concise; link to the shared skill source files
|
- Keep docs concise; link to the actual skill
|
||||||
|
source files (under `.claude/skills/`, etc.)
|
||||||
rather than duplicating content
|
rather than duplicating content
|
||||||
- Keep skill bodies harness-neutral; isolate slash-command
|
|
||||||
syntax and permissions in each harness integration
|
|
||||||
|
|
|
||||||
|
|
@ -1,17 +1,15 @@
|
||||||
# Claude Code Integration
|
# Claude Code Integration
|
||||||
|
|
||||||
[Claude Code](https://github.com/anthropics/claude-code)
|
[Claude Code](https://github.com/anthropics/claude-code)
|
||||||
integration for piker's shared coding-harness skills.
|
skills and workflows for piker development.
|
||||||
|
|
||||||
## Skills
|
## Skills
|
||||||
|
|
||||||
| Skill | Invocable | Description |
|
| Skill | Invocable | Description |
|
||||||
|-------|-----------|-------------|
|
|-------|-----------|-------------|
|
||||||
| [`commit-msg`](#commit-msg) | `/commit-msg` | Generate piker-style commit messages |
|
| [`commit-msg`](#commit-msg) | `/commit-msg` | Generate piker-style commit messages |
|
||||||
| `piker-conc-expert` | auto | Actor-tree and structured-concurrency invariants |
|
|
||||||
| `piker-profiling` | auto | `Profiler` API patterns for perf work |
|
| `piker-profiling` | auto | `Profiler` API patterns for perf work |
|
||||||
| `piker-slang` | auto | Communication style + slang guide |
|
| `piker-slang` | auto | Communication style + slang guide |
|
||||||
| `py-codestyle` | auto | Python source conventions |
|
|
||||||
| `pyqtgraph-optimization` | auto | Batch rendering patterns |
|
| `pyqtgraph-optimization` | auto | Batch rendering patterns |
|
||||||
| `timeseries-optimization` | auto | NumPy/Polars perf patterns |
|
| `timeseries-optimization` | auto | NumPy/Polars perf patterns |
|
||||||
|
|
||||||
|
|
@ -19,10 +17,8 @@ Skills marked **auto** are background knowledge
|
||||||
applied automatically when Claude detects relevance.
|
applied automatically when Claude detects relevance.
|
||||||
Only `commit-msg` is user-invoked via slash command.
|
Only `commit-msg` is user-invoked via slash command.
|
||||||
|
|
||||||
Portable skill source files live under
|
Skill source files live under
|
||||||
`.claude/skills/<skill-name>/SKILL.md` and are shared with
|
`.claude/skills/<skill-name>/SKILL.md`.
|
||||||
OpenCode. Claude-specific behavior belongs in command or
|
|
||||||
settings files, not the shared skill bodies.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -88,7 +84,8 @@ description context. Examples:
|
||||||
**Footer** (always):
|
**Footer** (always):
|
||||||
```
|
```
|
||||||
(this patch was generated in some part by
|
(this patch was generated in some part by
|
||||||
`claude-code` using `<model>` (`<provider>`))
|
[`claude-code`][claude-code-gh])
|
||||||
|
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||||
```
|
```
|
||||||
|
|
||||||
### Output Files
|
### Output Files
|
||||||
|
|
@ -97,8 +94,7 @@ After generation, the commit message is written to:
|
||||||
|
|
||||||
```
|
```
|
||||||
.claude/
|
.claude/
|
||||||
├── skills/commit-msg/msgs/
|
├── <timestamp>_<hash>_commit_msg.md # archived
|
||||||
│ └── <timestamp>_<hash>_commit_msg.md
|
|
||||||
└── git_commit_msg_LATEST.md # latest
|
└── git_commit_msg_LATEST.md # latest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -106,10 +102,18 @@ Where `<timestamp>` is ISO-8601 with seconds and
|
||||||
`<hash>` is the first 7 chars of the current
|
`<hash>` is the first 7 chars of the current
|
||||||
`HEAD` commit.
|
`HEAD` commit.
|
||||||
|
|
||||||
Use the latest file with an editor review before commit:
|
Use the latest file to feed into `git commit`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git commit --edit --file .claude/git_commit_msg_LATEST.md
|
git commit -F .claude/git_commit_msg_LATEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Or review/edit before committing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat .claude/git_commit_msg_LATEST.md
|
||||||
|
# edit if needed, then:
|
||||||
|
git commit -F .claude/git_commit_msg_LATEST.md
|
||||||
```
|
```
|
||||||
|
|
||||||
### Examples
|
### Examples
|
||||||
|
|
@ -124,32 +128,56 @@ Add `MktPair.fqme` property for symbol resolution
|
||||||
Factor `.claude/skills/` into proper subdirs
|
Factor `.claude/skills/` into proper subdirs
|
||||||
|
|
||||||
Deats,
|
Deats,
|
||||||
- use portable Agent Skills frontmatter
|
- `commit_msg/` -> `commit-msg/` w/ enhanced
|
||||||
- keep harness commands as thin delegates
|
frontmatter
|
||||||
- share supporting references across harnesses
|
- all background skills set `user-invocable: false`
|
||||||
|
- content split into supporting files
|
||||||
|
|
||||||
(this patch was generated in some part by
|
(this patch was generated in some part by
|
||||||
`claude-code` using `<model>` (`<provider>`))
|
[`claude-code`][claude-code-gh])
|
||||||
|
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||||
```
|
```
|
||||||
|
|
||||||
### Shared Frontmatter
|
### Frontmatter Reference
|
||||||
|
|
||||||
Skills use the Agent Skills fields understood by both
|
The skill's `SKILL.md` uses these Claude Code
|
||||||
Claude Code and OpenCode:
|
frontmatter fields:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
---
|
---
|
||||||
name: commit-msg
|
name: commit-msg
|
||||||
description: >
|
description: >
|
||||||
Generate piker-style git commit messages. Use when...
|
Generate piker-style git commit messages...
|
||||||
compatibility: Requires git.
|
argument-hint: "[optional-scope-or-description]"
|
||||||
metadata:
|
disable-model-invocation: true
|
||||||
author: goodboy
|
allowed-tools:
|
||||||
version: "1.0"
|
- Bash(git *)
|
||||||
|
- Read
|
||||||
|
- Grep
|
||||||
|
- Glob
|
||||||
|
- Write
|
||||||
---
|
---
|
||||||
```
|
```
|
||||||
|
|
||||||
Live git context is gathered by the skill at execution
|
| Field | Purpose |
|
||||||
time. Claude-only dynamic interpolation and tool-policy
|
|-------|---------|
|
||||||
frontmatter are intentionally excluded so another harness
|
| `argument-hint` | Shows hint in autocomplete |
|
||||||
can execute the same workflow.
|
| `disable-model-invocation` | Only user can trigger via `/commit-msg` |
|
||||||
|
| `allowed-tools` | Tools the skill can use |
|
||||||
|
|
||||||
|
### Dynamic Context
|
||||||
|
|
||||||
|
The skill injects live data at invocation time
|
||||||
|
via `!`backtick`` syntax in the `SKILL.md`:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## Current staged changes
|
||||||
|
!`git diff --staged --stat`
|
||||||
|
|
||||||
|
## Recent commit style reference
|
||||||
|
!`git log --oneline -10`
|
||||||
|
```
|
||||||
|
|
||||||
|
This means the staged diff stats and recent log
|
||||||
|
are always fresh when the skill runs -- no stale
|
||||||
|
context.
|
||||||
|
|
|
||||||
|
|
@ -1,143 +0,0 @@
|
||||||
# Split `brokerd.<broker>` into trading-only `brokerd` + new `datad.<broker>`
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
Today a single `brokerd.<broker>` actor hosts BOTH concerns:
|
|
||||||
|
|
||||||
- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers/<backend>/feed.py`/`symbols.py`,
|
|
||||||
- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers/<backend>/broker.py`, driven by `emsd`.
|
|
||||||
|
|
||||||
The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split.
|
|
||||||
|
|
||||||
**User-decided constraints:**
|
|
||||||
|
|
||||||
- `datad.<broker>` is a **sibling** of `brokerd.<broker>` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`).
|
|
||||||
- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working.
|
|
||||||
- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes.
|
|
||||||
|
|
||||||
## Target topology
|
|
||||||
|
|
||||||
```
|
|
||||||
pikerd
|
|
||||||
├── datad.ib feed bus, shm writers, tsp history, symbol search
|
|
||||||
├── brokerd.ib open_trade_dialog only (EMS-spawned, lazy)
|
|
||||||
├── emsd
|
|
||||||
│ └── paperboi.ib (paper mode; opens its feed via datad)
|
|
||||||
└── samplerd
|
|
||||||
```
|
|
||||||
|
|
||||||
## Key verified facts (load-bearing)
|
|
||||||
|
|
||||||
- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor.
|
|
||||||
- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree.
|
|
||||||
- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param.
|
|
||||||
- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS.
|
|
||||||
- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns).
|
|
||||||
- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`.
|
|
||||||
- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split.
|
|
||||||
- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves.
|
|
||||||
- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad.
|
|
||||||
|
|
||||||
## Module decision
|
|
||||||
|
|
||||||
New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Stage 0 — prep: backend module grouping + ep introspection (no topology change)
|
|
||||||
|
|
||||||
1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks.
|
|
||||||
2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged:
|
|
||||||
- `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']`
|
|
||||||
- `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']`
|
|
||||||
- `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']`
|
|
||||||
- `ib`: adjust existing groups so each includes `'api'`
|
|
||||||
- flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior).
|
|
||||||
3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`).
|
|
||||||
|
|
||||||
**Gate**: full pytest green, zero behavior change.
|
|
||||||
|
|
||||||
## Stage 1 — introduce `datad` daemon machinery (additive)
|
|
||||||
|
|
||||||
New `piker/data/_daemon.py`:
|
|
||||||
|
|
||||||
- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`.
|
|
||||||
- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`.
|
|
||||||
- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`.
|
|
||||||
- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`.
|
|
||||||
- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`).
|
|
||||||
|
|
||||||
Supporting edits:
|
|
||||||
|
|
||||||
- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`).
|
|
||||||
- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`).
|
|
||||||
- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.)
|
|
||||||
|
|
||||||
**Gate**: suite green + new spawn test; nothing routes through datad yet.
|
|
||||||
|
|
||||||
## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`)
|
|
||||||
|
|
||||||
Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal).
|
|
||||||
|
|
||||||
`piker/clearing/_ems.py`:
|
|
||||||
|
|
||||||
1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it.
|
|
||||||
2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`).
|
|
||||||
3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`.
|
|
||||||
|
|
||||||
Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry.
|
|
||||||
|
|
||||||
**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke.
|
|
||||||
|
|
||||||
## Stage 3 — feed layer hard cutover to datad
|
|
||||||
|
|
||||||
1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep.
|
|
||||||
2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.)
|
|
||||||
3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`:
|
|
||||||
- `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`)
|
|
||||||
- `piker/brokers/core.py:33,175` (`symbol_search`)
|
|
||||||
- `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`)
|
|
||||||
- `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'`
|
|
||||||
4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`.
|
|
||||||
|
|
||||||
**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.<broker>` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane.
|
|
||||||
|
|
||||||
## Stage 4 — caps-sec slimming, validation, ib client-id
|
|
||||||
|
|
||||||
1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all.
|
|
||||||
2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously.
|
|
||||||
3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys.
|
|
||||||
4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring.
|
|
||||||
|
|
||||||
**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Risk register
|
|
||||||
|
|
||||||
| Risk | Sev | Mitigation |
|
|
||||||
|---|---|---|
|
|
||||||
| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns |
|
|
||||||
| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test |
|
|
||||||
| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast |
|
|
||||||
| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only |
|
|
||||||
| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it |
|
|
||||||
| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly |
|
|
||||||
|
|
||||||
## Verification (per stage gates above, plus end-to-end)
|
|
||||||
|
|
||||||
- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed).
|
|
||||||
- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger <broker>.paper`.
|
|
||||||
- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4).
|
|
||||||
|
|
||||||
## Critical files
|
|
||||||
|
|
||||||
- **new** `piker/data/_daemon.py` — all datad machinery
|
|
||||||
- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn
|
|
||||||
- `piker/data/feed.py` — spawn cutover + actor-name assert
|
|
||||||
- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`)
|
|
||||||
- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports
|
|
||||||
- `piker/data/validate.py` — `get_eps()` helper
|
|
||||||
- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods`
|
|
||||||
- `piker/accounting/cli.py` — keyword-form dialog call
|
|
||||||
- `tests/test_services.py` — inverted + new assertions
|
|
||||||
|
|
@ -1,51 +0,0 @@
|
||||||
# OpenCode Integration
|
|
||||||
|
|
||||||
[OpenCode](https://opencode.ai/) uses piker's shared
|
|
||||||
repo-specific skills directly from `.claude/skills/`.
|
|
||||||
There is no second copy of each skill to keep in sync.
|
|
||||||
|
|
||||||
## Available Skills
|
|
||||||
|
|
||||||
| Skill | Activation | Specialization |
|
|
||||||
|-------|------------|----------------|
|
|
||||||
| `commit-msg` | `/commit-msg` or explicit request | Piker commit-message style and artifacts |
|
|
||||||
| `piker-conc-expert` | automatic | `tractor` actor topology, RPC, and cancellation |
|
|
||||||
| `piker-profiling` | automatic | Cross-actor `Profiler` instrumentation |
|
|
||||||
| `piker-slang` | automatic | Project communication style |
|
|
||||||
| `py-codestyle` | automatic | Python source conventions |
|
|
||||||
| `pyqtgraph-optimization` | automatic | Batched Qt graphics rendering |
|
|
||||||
| `timeseries-optimization` | automatic | NumPy and Polars timeseries performance |
|
|
||||||
|
|
||||||
OpenCode's project command lives at
|
|
||||||
`.opencode/commands/commit-msg.md`. It delegates to the
|
|
||||||
shared `commit-msg` skill and only supplies command
|
|
||||||
arguments; workflow logic remains in the skill.
|
|
||||||
|
|
||||||
A user-local skill with the same name may override a repo
|
|
||||||
skill. Treat `opencode debug skill` as authoritative when
|
|
||||||
diagnosing which source is active.
|
|
||||||
|
|
||||||
## Verify Discovery
|
|
||||||
|
|
||||||
From the repository root:
|
|
||||||
|
|
||||||
```console
|
|
||||||
opencode debug skill
|
|
||||||
opencode debug config
|
|
||||||
```
|
|
||||||
|
|
||||||
`opencode debug skill` should list the piker skills from
|
|
||||||
the project checkout. Restart OpenCode after changing a
|
|
||||||
skill or command because configuration-time files are not
|
|
||||||
hot-reloaded into an active session.
|
|
||||||
|
|
||||||
## Portability Rules
|
|
||||||
|
|
||||||
- Use only portable Agent Skills frontmatter in shared
|
|
||||||
`SKILL.md` files.
|
|
||||||
- Keep harness-specific command syntax under
|
|
||||||
`.opencode/commands/`.
|
|
||||||
- Detect the active harness and model at runtime; do not
|
|
||||||
hardcode Claude Code attribution in shared workflows.
|
|
||||||
- Keep supporting references beside each shared skill so
|
|
||||||
relative links resolve in every harness.
|
|
||||||
|
|
@ -1,59 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:08:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Session-initiating instruction (driving all 7 commits in
|
|
||||||
this series):
|
|
||||||
|
|
||||||
> ok i want you to become the distributed runtime and
|
|
||||||
> concurrency expert for this project - namely acquire
|
|
||||||
> a deep understanding of tractor and how it's used.
|
|
||||||
> then i want you to attempt to factor our current
|
|
||||||
> brokerd service daemon into 2 daemons: a brokerd
|
|
||||||
> which only hosts live/paper trading endpoint tasks
|
|
||||||
> [..] a new `datad` subdaemon which in a separate
|
|
||||||
> subactor serves all the data feed service tasks [..]
|
|
||||||
> give me a mega detailed plan on how to approach this,
|
|
||||||
> and a staged approach for the implementation.
|
|
||||||
|
|
||||||
Proximate driver for THIS commit: during the approved
|
|
||||||
plan's stage-0 test gate the agent discovered test
|
|
||||||
subactors were writing into the user's REAL
|
|
||||||
`~/.config/piker/accounting/` files (a bogus paper fill
|
|
||||||
landed in `account.kraken.paper.toml`) because the old
|
|
||||||
test-dir override in `config.get_app_dir()` was
|
|
||||||
commented out and could never work in spawned
|
|
||||||
subactors anyway. Fix applied autonomously under the
|
|
||||||
approved plan; no explicit per-fix user prompt.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Restore `pytest` config-dir isolation by resolving the
|
|
||||||
per-test tmp dir lazily (at conf-path access time) from
|
|
||||||
`tractor` runtime-vars, which propagate down the actor
|
|
||||||
tree; route all conf-path derivation through
|
|
||||||
`config.get_conf_dir()` so the override is effective in
|
|
||||||
every (sub)actor.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/config.py` — add `_maybe_use_test_dir()`;
|
|
||||||
hook in `get_conf_dir()`; route `get_conf_path()` +
|
|
||||||
`load()` mkdir through it
|
|
||||||
- `piker/accounting/_ledger.py` — derive ledger dir
|
|
||||||
via `config.get_conf_dir()` (not the global)
|
|
||||||
- `piker/accounting/_pos.py` — same for account files
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:08:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/config.py`
|
|
||||||
|
|
||||||
Generated: `config._maybe_use_test_dir()` — lazily
|
|
||||||
reads `piker_test_dir` from
|
|
||||||
`tractor.runtime._state._runtime_vars['piker_vars']`
|
|
||||||
(pre-loaded by `open_piker_runtime()` from the
|
|
||||||
`tests.conftest._open_test_pikerd()` overrides) and
|
|
||||||
calls `_override_config_dir()` when set. Hooked at the
|
|
||||||
top of `get_conf_dir()`; `get_conf_path()` and
|
|
||||||
`load()`'s dir-creation rerouted through
|
|
||||||
`get_conf_dir()`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/accounting/_ledger.py`
|
|
||||||
> `git log -1 -p --follow -- piker/accounting/_pos.py`
|
|
||||||
|
|
||||||
Generated: ledger/account dir derivation switched from
|
|
||||||
the `config._config_dir` module global to
|
|
||||||
`config.get_conf_dir()` + `mkdir(parents=True,
|
|
||||||
exist_ok=True)` for nested tmp-dir creation.
|
|
||||||
|
|
||||||
Key diagnostic reasoning (verbatim from session):
|
|
||||||
|
|
||||||
The old (commented-out) `get_app_dir()` override gated
|
|
||||||
on `'pytest' in sys.modules` which can NEVER work in
|
|
||||||
spawned subactors (fresh procs, no pytest import); as
|
|
||||||
a result test `paperboi`/daemon actors were writing
|
|
||||||
into the user's REAL `~/.config/piker/accounting/`
|
|
||||||
files. Evidence: `account.kraken.paper.toml` gained a
|
|
||||||
single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
|
|
||||||
(a test fill), and `trades_kraken_paper.toml` (252B)
|
|
||||||
contained only that fill. The resolution must be
|
|
||||||
checked lazily at config-path access time (NOT import
|
|
||||||
time) since sub-actors only receive runtime-vars once
|
|
||||||
their `tractor` runtime has fully booted.
|
|
||||||
|
|
@ -1,53 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:10:22Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
|
|
||||||
driver: the approved plan's per-stage test gates could
|
|
||||||
not run AT ALL — the branch base was broken vs.
|
|
||||||
`tractor` git `main` (`AttributeError: 'tuple' object
|
|
||||||
has no attribute 'unwrap'` at `pikerd` boot, stale
|
|
||||||
`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
|
|
||||||
fixed forward autonomously to (re)establish the gate
|
|
||||||
baseline, continuing the `repair_tests` branch lineage
|
|
||||||
(verified already merged in ancestry via `git cherry`
|
|
||||||
during a user-requested branch-overlap survey).
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Port the service layer + test suites to current
|
|
||||||
`tractor` APIs: addr-type normalization in
|
|
||||||
`open_pikerd()`, `query_actor()`/`wait_for_actor()`
|
|
||||||
kwarg renames, modern self-cancel absorption semantics
|
|
||||||
in the cancel-method test harness, exc/position attr
|
|
||||||
renames, a paper-EMS startup-budget bump and a syntax
|
|
||||||
fix in `.deribit.api`.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/service/_actor_runtime.py` — `wrap_address()`
|
|
||||||
normalize before `.unwrap()` in `open_pikerd()`
|
|
||||||
- `piker/service/_registry.py` — `check_for_service()`
|
|
||||||
-> `query_actor(regaddr=)` + 2-tuple yield +
|
|
||||||
`open_registry(addrs=)`
|
|
||||||
- `piker/brokers/deribit/api.py` — missing comma in
|
|
||||||
`tractor.trionics` import tuple
|
|
||||||
- `tests/test_services.py` — `registry_addr=` kwarg,
|
|
||||||
raddr unwraps, cancel-semantics harness rewrite,
|
|
||||||
`fail_after` 9 -> 19s
|
|
||||||
- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,56 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:10:22Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
|
||||||
|
|
||||||
Generated: normalize each registry addr via
|
|
||||||
`tractor.discovery._addr.wrap_address()` before
|
|
||||||
`.unwrap()`-ing for the `accept_addrs` bind check —
|
|
||||||
entries may be raw `tuple`s when passed in from (test)
|
|
||||||
client code. Import-path precedent taken from
|
|
||||||
`piker/cli/__init__.py:336`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/service/_registry.py`
|
|
||||||
|
|
||||||
Generated: `check_for_service()` ported to
|
|
||||||
`tractor.query_actor(name, regaddr=...)` (kwarg was
|
|
||||||
`arbiter_sockaddr=`), unpacking the new
|
|
||||||
`(sockaddr, portal)` yield, and passing the
|
|
||||||
now-required `open_registry(addrs=Registry.addrs)`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- tests/test_services.py`
|
|
||||||
> `git log -1 -p --follow -- tests/test_ems.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
|
|
||||||
|
|
||||||
Key diagnostic reasoning (verbatim from session):
|
|
||||||
|
|
||||||
- the "DID NOT RAISE ContextCancelled" failure: in this
|
|
||||||
test the client actor IS pikerd (in-proc), and
|
|
||||||
current `tractor` main absorbs a `ContextCancelled`
|
|
||||||
whose canceller is your own actor — self-requested
|
|
||||||
cancels now exit cleanly instead of raising; the
|
|
||||||
'sigint' variant propagates a bare collapsed
|
|
||||||
`KeyboardInterrupt` rather than a
|
|
||||||
`BaseExceptionGroup`.
|
|
||||||
- the hard-coded `trio.fail_after(9)` startup budget is
|
|
||||||
marginal — full stack boot (pikerd -> emsd ->
|
|
||||||
brokerd.kraken -> paperboi + live kraken symbology
|
|
||||||
fetch) occasionally exceeds 9s -> bumped to 19s.
|
|
||||||
- `RemoteActorError.type` -> `.boxed_type`;
|
|
||||||
`Position.size` -> `.cumsize` (the paper engine
|
|
||||||
populates `BrokerdPosition.size` from `pp.cumsize`).
|
|
||||||
- overlap survey (user-requested): all of the
|
|
||||||
`repair_tests` branch commits are already in this
|
|
||||||
stack's ancestry; this commit finishes that branch's
|
|
||||||
port mission (its f4c4f1e2 fixed `conftest.py`'s
|
|
||||||
`arbiter_sockaddr` usage; this fixes the remaining
|
|
||||||
`test_services.py` + `check_for_service()` sites).
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:11:05Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171105Z_bc6e18d7_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 0" prep. Plan-shaping user
|
|
||||||
decisions captured via in-session Q&A:
|
|
||||||
|
|
||||||
- `datad.<broker>` topology: sibling of
|
|
||||||
`brokerd.<broker>` under `pikerd` (vs. child
|
|
||||||
subactor).
|
|
||||||
- migration: hard cutover, staged by layer (no
|
|
||||||
dual-mode runtime flag).
|
|
||||||
- post-split `brokerd` scope: trading-only +
|
|
||||||
EMS-lazy-spawned (charts/CLI never touch it).
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Declare per-daemon-kind backend submod groups
|
|
||||||
(`_datad_mods`/`_brokerd_mods`) in the split-style
|
|
||||||
backends, keyed to the pre-existing
|
|
||||||
`piker.data.validate._eps` contract; add a
|
|
||||||
`validate.get_eps()` introspection helper; ws-import
|
|
||||||
hygiene in `.kraken.broker`. Zero behavior change
|
|
||||||
(`__enable_modules__` unions unchanged).
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/validate.py` — add `get_eps()`
|
|
||||||
- `piker/brokers/kraken/__init__.py` — mod groups
|
|
||||||
- `piker/brokers/binance/__init__.py` — mod groups
|
|
||||||
(note: no `symbols.py`; search eps live in `.feed`)
|
|
||||||
- `piker/brokers/deribit/__init__.py` — datad-only
|
|
||||||
groups (no `broker.py` yet)
|
|
||||||
- `piker/brokers/ib/__init__.py` — add `'api'` to the
|
|
||||||
pre-existing `_datad_mods`
|
|
||||||
- `piker/brokers/kraken/broker.py` — import
|
|
||||||
`NoBsWs`/`open_autorecon_ws` from
|
|
||||||
`piker.data._web_bs` directly
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:11:05Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/data/validate.py`
|
|
||||||
|
|
||||||
Generated: `get_eps(mod, kind) -> dict[str, Callable]`
|
|
||||||
returning the daemon-kind's ep funcs defined by a
|
|
||||||
backend mod, keyed by ep name, missing eps excluded;
|
|
||||||
sourced from the existing `_eps` grouping table
|
|
||||||
(`'middleware' | 'datad' | 'brokerd'`).
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/kraken/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/binance/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/deribit/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/kraken/broker.py`
|
|
||||||
|
|
||||||
Key exploration findings driving the design (from the
|
|
||||||
planning phase, 3 parallel explore agents + 1 architect
|
|
||||||
agent):
|
|
||||||
|
|
||||||
- the codebase already anticipated this split:
|
|
||||||
`validate._eps` groups eps into 'datad' vs 'brokerd'
|
|
||||||
kinds and `ib/__init__.py` already declared
|
|
||||||
`_brokerd_mods`/`_datad_mods`; `brokers/_daemon.py`
|
|
||||||
carried the literal TODO "rename the daemon to datad
|
|
||||||
prolly once we split up broker vs. data tasks into
|
|
||||||
separate actors?".
|
|
||||||
- the feared kraken feed<->broker "shared ws state"
|
|
||||||
hazard is mild: `NoBsWs`/`open_autorecon_ws`
|
|
||||||
originate in `piker.data._web_bs` (re-exported via
|
|
||||||
`.kraken.feed`); only `stream_messages` is feed-local
|
|
||||||
and it's a pure parser — in-process imports only,
|
|
||||||
each actor opens its own ws conn.
|
|
||||||
- `enable_modules` gates RPC entry, NOT python imports,
|
|
||||||
so backend trading mods may keep importing feed-side
|
|
||||||
helpers in-process post-split.
|
|
||||||
|
|
@ -1,47 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:11:42Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171142Z_119d2c04_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 1": introduce the `datad` daemon
|
|
||||||
machinery additively (nothing routes through it yet).
|
|
||||||
User-decided constraint applied: `datad.<broker>` is a
|
|
||||||
SIBLING of `brokerd.<broker>` under `pikerd`, spawned
|
|
||||||
via the existing `Services` + `maybe_spawn_daemon()`
|
|
||||||
machinery.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
New `piker/data/_daemon.py` hosting the
|
|
||||||
`datad.<broker>` feed-only daemon-actor: lifetime
|
|
||||||
fixture owning the actor-global `_FeedsBus`, init/spawn
|
|
||||||
fns mirroring `.brokers._daemon` conventions and the
|
|
||||||
`samplerd` sub-daemon precedent, plus root-mod
|
|
||||||
registration, `piker.service` re-exports and a spawn
|
|
||||||
test.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/_daemon.py` — NEW:
|
|
||||||
`_setup_persistent_datad()`, `datad_init()`,
|
|
||||||
`spawn_datad()`, `maybe_spawn_datad()`,
|
|
||||||
`_datad_service_mods`
|
|
||||||
- `piker/service/_actor_runtime.py` — add
|
|
||||||
`piker.data._daemon` to `_root_modules`
|
|
||||||
- `piker/service/__init__.py` — re-export spawn eps
|
|
||||||
- `tests/test_services.py` — add `test_datad_spawn`
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,60 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:11:42Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/data/_daemon.py`
|
|
||||||
|
|
||||||
Generated symbols + key design decisions:
|
|
||||||
|
|
||||||
- `_datad_service_mods: list[str]` — datad-always
|
|
||||||
enabled mods, the data-side successor to the old
|
|
||||||
`piker.brokers._daemon._data_mods` set; kept minimal
|
|
||||||
per the caps-sec model.
|
|
||||||
- `_setup_persistent_datad()` — `@tractor.context`
|
|
||||||
lifetime fixture: console-log setup then allocates
|
|
||||||
the actor-global feed bus via
|
|
||||||
`feed.get_feed_bus(brokername, service_nursery)`
|
|
||||||
exactly as the old brokerd fixture did, pinned open
|
|
||||||
with `ctx.started()` + `sleep_forever()`.
|
|
||||||
- `datad_init()` — actor name `f'datad.{brokername}'`;
|
|
||||||
copies backend `_spawn_kwargs` (CRITICAL for `ib`'s
|
|
||||||
`infect_asyncio=True`); builds `enable_modules` from
|
|
||||||
`getattr(brokermod, '_datad_mods',
|
|
||||||
getattr(brokermod, '__enable_modules__', []))` as
|
|
||||||
the flat-backend fallback.
|
|
||||||
- `spawn_datad()` — `Services.actor_n.start_actor()` +
|
|
||||||
`Services.start_service_task()` exactly mirroring
|
|
||||||
`spawn_brokerd()`; dedup-composes enable mods via
|
|
||||||
`list(dict.fromkeys(...))`.
|
|
||||||
- `maybe_spawn_datad()` — wraps `maybe_spawn_daemon(
|
|
||||||
service_name=f'datad.{brokername}', ...)`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
|
||||||
> `git log -1 -p --follow -- piker/service/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- tests/test_services.py`
|
|
||||||
|
|
||||||
Design rationale (verbatim from session):
|
|
||||||
|
|
||||||
- `_root_modules` must gain `piker.data._daemon` so
|
|
||||||
`pikerd_portal.run(spawn_datad, ...)` resolves in
|
|
||||||
the root.
|
|
||||||
- the `Services`-based impl style deliberately mirrors
|
|
||||||
`spawn_brokerd()` so the eventual `tractor.hilevel`
|
|
||||||
`ServiceMngr` port (see the `service_mng_to_tractor`
|
|
||||||
branch's d8c21d44 prep, surfaced by the
|
|
||||||
user-requested branch-overlap survey) lands
|
|
||||||
symmetrically on both spawn fns.
|
|
||||||
- mod placement (`piker/data/_daemon.py` vs.
|
|
||||||
generalizing `piker.brokers._daemon`) follows the
|
|
||||||
per-subsystem daemon-mod convention
|
|
||||||
(`.clearing._ems`, `.data._sampling`) and resolves
|
|
||||||
the existing TODO at `brokers/_daemon.py:49` ("move
|
|
||||||
this def to the `.data` subpkg").
|
|
||||||
|
|
@ -1,47 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:12:26Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171226Z_64181219_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 2": decouple the clearing layer
|
|
||||||
from `feed.portals` BEFORE the feed cutover so live
|
|
||||||
trading works at every stage boundary. User-decided
|
|
||||||
constraint applied: post-split `brokerd` is
|
|
||||||
trading-only and spawned LAZILY only by `emsd`'s
|
|
||||||
`open_brokerd_dialog()` path.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Kill the single coupling forcing feed + trading eps
|
|
||||||
into one actor (`Router.open_trade_relays()` pulling
|
|
||||||
its trades portal from `feed.portals[brokermod]`):
|
|
||||||
`open_brokerd_dialog()` now (maybe) spawns/finds
|
|
||||||
`brokerd.<broker>` itself and ONLY when a live
|
|
||||||
trades-ep will actually open; paper mode never touches
|
|
||||||
it. Pre-cutover this is a pure refactor (registry
|
|
||||||
lookup finds the same feed-spawned daemon).
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/clearing/_ems.py` — `open_brokerd_dialog()`
|
|
||||||
re-sig + inner `acquire_live_portal()`;
|
|
||||||
`Router.maybe_open_brokerd_dialog()` drops `portal`
|
|
||||||
param; `open_trade_relays()` drops the
|
|
||||||
`feed.portals` lookup
|
|
||||||
- `piker/accounting/cli.py` — keyword-form `portal=`
|
|
||||||
override kept for the `piker ledger` ad-hoc actor
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:12:26Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/clearing/_ems.py`
|
|
||||||
|
|
||||||
Generated: new `open_brokerd_dialog()` signature
|
|
||||||
`(brokermod, exec_mode, fqme=None, portal=None,
|
|
||||||
loglevel=None)` with an inner `@acm
|
|
||||||
acquire_live_portal()` that yields the caller-provided
|
|
||||||
`portal` override (the `piker ledger` path) else
|
|
||||||
`maybe_spawn_brokerd(brokermod.name)` — designated THE
|
|
||||||
one place a live, credentialed `brokerd.<broker>` gets
|
|
||||||
booted post-split. The eager
|
|
||||||
`portal.open_context(trades_endpoint, ...)`
|
|
||||||
construction moved inside that block so the paper
|
|
||||||
short-circuit never acquires a live portal.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/accounting/cli.py`
|
|
||||||
|
|
||||||
Key analysis (verbatim from session):
|
|
||||||
|
|
||||||
- `feed.portals` had exactly ONE trading consumer:
|
|
||||||
`piker/clearing/_ems.py:671` (`portal =
|
|
||||||
feed.portals[brokermod]`) — the single coupling
|
|
||||||
forcing feed + trading into one actor.
|
|
||||||
- `piker ledger` (`accounting/cli.py:100-157`) is a
|
|
||||||
hidden consumer: it calls `broker_init()` directly,
|
|
||||||
spawns its own ad-hoc actor and passes the portal
|
|
||||||
positionally — the new signature must keep an
|
|
||||||
explicit `portal:` override param for this path.
|
|
||||||
- stage sequencing: clearing decouple lands BEFORE the
|
|
||||||
feed cutover, otherwise `feed.portals` would hand
|
|
||||||
emsd a `datad` portal and live `open_trade_dialog`
|
|
||||||
RPC would fail.
|
|
||||||
|
|
@ -1,48 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:12:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171259Z_59d5d9a6_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 3": the feed-layer hard cutover
|
|
||||||
(user-decided: no dual-mode runtime flag, each stage
|
|
||||||
lands fully working).
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
The topology flip: all data-feed consumers route to
|
|
||||||
`datad.<broker>`; the `brokerd` lifetime fixture slims
|
|
||||||
to trading-only; UI/CLI data-flavoured spawn sites
|
|
||||||
repointed; test expectations inverted incl. an explicit
|
|
||||||
negative assert that paper sessions spawn NO
|
|
||||||
`brokerd.<broker>`.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/feed.py` — `open_feed()` ->
|
|
||||||
`maybe_spawn_datad()`; `open_feed_bus()` actor-name
|
|
||||||
assert -> `'datad'`; `install_brokerd_search` ->
|
|
||||||
`install_datad_search`; comment sweep
|
|
||||||
- `piker/brokers/_daemon.py` —
|
|
||||||
`_setup_persistent_brokerd()` slimmed (feed-bus
|
|
||||||
alloc moved to the datad fixture)
|
|
||||||
- `piker/ui/_app.py`, `piker/ui/cli.py`,
|
|
||||||
`piker/brokers/core.py`, `piker/brokers/cli.py`,
|
|
||||||
`piker/ui/kivy/option_chain.py` — spawn-site
|
|
||||||
repoints
|
|
||||||
- `tests/test_services.py` — datad assertions + the
|
|
||||||
no-`brokerd.kraken` negative check
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:12:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/data/feed.py`
|
|
||||||
|
|
||||||
Generated: `open_feed()` spawn cutover with the
|
|
||||||
`maybe_spawn_datad` import done "relative-direct" from
|
|
||||||
`._daemon` — NOT via `piker.service` — to dodge a
|
|
||||||
partial-init cycle: when `piker.data.feed` loads as
|
|
||||||
part of `piker.service.__init__` executing its own
|
|
||||||
`piker.data._daemon` import, the service pkg is
|
|
||||||
mid-init and its `maybe_spawn_datad` binding does not
|
|
||||||
exist yet. The `open_feed_bus()` local-state sanity
|
|
||||||
assert flips `'brokerd' in servicename` ->
|
|
||||||
`'datad' in servicename` (the only actor-name assert
|
|
||||||
in the tree, verified by grep).
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
|
||||||
|
|
||||||
Generated: `_setup_persistent_brokerd()` slimmed to
|
|
||||||
console-log setup + pinned-open ctx; drops the bus
|
|
||||||
alloc, the `assert not feed._bus`, the service nursery
|
|
||||||
(backend `open_trade_dialog()` ctxs own their task
|
|
||||||
trees) and the `eg.ExceptionGroup` handler. The
|
|
||||||
`piker ledger` ad-hoc actor enters this same slimmed
|
|
||||||
fixture — exactly what it needs.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/ui/_app.py`
|
|
||||||
> `git log -1 -p --follow -- piker/ui/cli.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/core.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/cli.py`
|
|
||||||
> `git log -1 -p --follow -- piker/ui/kivy/option_chain.py`
|
|
||||||
> `git log -1 -p --follow -- tests/test_services.py`
|
|
||||||
|
|
||||||
Verification (from session): per-suite gates green at
|
|
||||||
this commit (services 5-passed incl. the new negative
|
|
||||||
assert, feeds 3-passed); a headless live smoke
|
|
||||||
(`maybe_open_pikerd` + `open_feed(['xbtusdt.kraken'])`
|
|
||||||
on an alt registry port) confirmed quotes flowing via
|
|
||||||
`datad.kraken` + `samplerd` with `check_for_service(
|
|
||||||
'brokerd.kraken') is None`. Known pre-existing flake
|
|
||||||
documented: `test_multi_fill_positions`' second
|
|
||||||
in-proc runtime boot wedges ~50% (zombie subactor w/
|
|
||||||
unread parent-IPC bytes); reproduced with the split
|
|
||||||
fully reverted so NOT a regression of this work.
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:13:44Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171344Z_eee19de0_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's final "stage 4": caps-sec slimming of
|
|
||||||
the (live, credentialed) trading actor + the `ib`
|
|
||||||
dual-daemon `client_id` collision mitigation flagged in
|
|
||||||
the plan's risk register.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
`brokerd` loses ALL `piker.data.*` (feed) RPC mods;
|
|
||||||
spawn fails fast for datad-only backends with a "use
|
|
||||||
paper-mode" error; `ib`'s default api-gw `client_id`
|
|
||||||
gets a per-daemon-kind offset so `datad.ib` +
|
|
||||||
`brokerd.ib` don't collide on connect.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/_daemon.py` — `_data_mods` -> minimal
|
|
||||||
`_brokerd_service_mods`; `broker_init()` reads
|
|
||||||
`_brokerd_mods` (fallback `__enable_modules__`);
|
|
||||||
`spawn_brokerd()` fail-fast via `validate.get_eps()`
|
|
||||||
- `piker/brokers/ib/api.py` — role-based `client_id`
|
|
||||||
offset in `load_aio_clients()`
|
|
||||||
- `piker/cli/__init__.py` — resolved "expose datad"
|
|
||||||
TODO
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:13:44Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
|
||||||
|
|
||||||
Generated: the fail-fast originally landed in
|
|
||||||
`broker_init()` but was relocated to `spawn_brokerd()`
|
|
||||||
mid-implementation after realizing `piker ledger`
|
|
||||||
calls `broker_init()` directly even for paper accounts
|
|
||||||
on datad-only backends (would have crashed the cli);
|
|
||||||
the service-spawn path is the correct enforcement
|
|
||||||
seam. Error text:
|
|
||||||
|
|
||||||
Backend 'kucoin' offers NO `brokerd` (live
|
|
||||||
order-control) eps!? It is likely a datad-only
|
|
||||||
provider, use paper-mode for clearing instead.
|
|
||||||
|
|
||||||
(verified live via a `trio.run()` unit check.)
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
|
||||||
|
|
||||||
Generated: in `load_aio_clients()`, when `client_id`
|
|
||||||
is the 6116 default: `datad`-named actors offset +16
|
|
||||||
(disjoint from `brokerd`'s linear `client_id + i`
|
|
||||||
retry range), other non-`brokerd` (ad-hoc test/cli)
|
|
||||||
actors +32. Rationale from the plan's risk register:
|
|
||||||
post-split BOTH per-broker daemons connect to the same
|
|
||||||
TWS/gw endpoint; a shared default id collides and
|
|
||||||
burns up to `connect_timeout * retries` (90s) in
|
|
||||||
retry cycles.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/cli/__init__.py`
|
|
||||||
|
|
@ -1,66 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:33:09Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
|
||||||
scope: docs
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The session-initiating instruction, verbatim (this doc
|
|
||||||
IS its "mega detailed plan" deliverable):
|
|
||||||
|
|
||||||
> ok i want you to become the distributed runtime and
|
|
||||||
> concurrency expert for this project - namely acquire
|
|
||||||
> a deep understanding of tractor and how it's used.
|
|
||||||
> then i want you to atttempt to factor our current
|
|
||||||
> brokerd service daemon into 2 daemons:
|
|
||||||
>
|
|
||||||
> - a brokerd which only hosts live/paper trading
|
|
||||||
> endoint tasks as defined namely within all
|
|
||||||
> piker/brokers/<backend>/broker.py mods
|
|
||||||
> - a new `datad` subdaemon which in a separate
|
|
||||||
> subactor serves all the data feed service tasks
|
|
||||||
> namely delivered by endpoints in
|
|
||||||
> piker/brokers/<backend>/feed.py (or similar for
|
|
||||||
> less mature backends) and as more rigorously
|
|
||||||
> defined by the validation machinery in
|
|
||||||
> piker.data.validate.
|
|
||||||
>
|
|
||||||
> give me a mega detailed plan on how to approach
|
|
||||||
> this, and a staged approach for the implementation.
|
|
||||||
|
|
||||||
Plan-shaping user decisions (in-session Q&A): sibling
|
|
||||||
topology under `pikerd`; hard cutover staged by layer;
|
|
||||||
trading-only EMS-lazy-spawned `brokerd`. The human
|
|
||||||
staged the doc copy into `ai/claude-code/plans/` and
|
|
||||||
requested this commit after the 7 implementation
|
|
||||||
commits landed.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
The staged design doc for the `brokerd` -> (`datad` +
|
|
||||||
`brokerd`) split: context, target supervision topology,
|
|
||||||
load-bearing verified facts, per-stage file-level
|
|
||||||
changes with gates, risk register and an end-to-end
|
|
||||||
verification matrix. Produced via 3 parallel explore
|
|
||||||
agents + 1 architect agent + human Q&A before any code
|
|
||||||
was written; all 7 implementation commits in this
|
|
||||||
branch reference its stages in their provenance
|
|
||||||
entries.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `ai/claude-code/plans/datad_service.md` — the design
|
|
||||||
plan doc (human-staged copy of the AI-authored plan)
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None to the doc content — committed as generated. NB:
|
|
||||||
the implementation deviated from the plan-as-written in
|
|
||||||
4 places, see the raw file's deviation log.
|
|
||||||
|
|
@ -1,51 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:33:09Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry; the AI output here is the
|
|
||||||
committed doc itself (a design plan, not code) so no
|
|
||||||
verbatim copy is duplicated:
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md`
|
|
||||||
|
|
||||||
## Implementation deviation log
|
|
||||||
|
|
||||||
Where the landed 7-commit series differs from the plan
|
|
||||||
as written (recorded for reviewer transparency; the
|
|
||||||
plan doc is committed unmodified):
|
|
||||||
|
|
||||||
1. Commit ordering: the `pytest` config-dir isolation
|
|
||||||
fix lands BEFORE the `tractor`-API drift port. The
|
|
||||||
plan's stage gates assumed a runnable+isolated test
|
|
||||||
baseline; per-commit gating exposed that without
|
|
||||||
isolation the paper-EMS test reads the user's real
|
|
||||||
(polluted) `account.kraken.paper.toml` and reddens.
|
|
||||||
2. The plan's "stage 0: full pytest green" gate
|
|
||||||
required first repairing pre-existing branch
|
|
||||||
breakage vs `tractor` git `main` (boot
|
|
||||||
`AttributeError`, stale discovery/exc/position
|
|
||||||
APIs) — that repair became its own commit
|
|
||||||
("Port service+tests to latest `tractor` APIs")
|
|
||||||
rather than plan-stage work.
|
|
||||||
3. Stage-4 fail-fast placement: the plan said
|
|
||||||
`broker_init()` raises on brokerd-ep-less backends;
|
|
||||||
implementation moved the raise to `spawn_brokerd()`
|
|
||||||
since `piker ledger` calls `broker_init()` directly
|
|
||||||
even for paper accounts on datad-only backends.
|
|
||||||
4. `brokers/_daemon.py` change grouping: the
|
|
||||||
import-cleanup hunks (`exceptiongroup`, `_FeedsBus`
|
|
||||||
type-only import) landed with the caps-sec slim
|
|
||||||
commit instead of the fixture-slim commit, keeping
|
|
||||||
each intermediate tree import-clean without
|
|
||||||
sub-hunk surgery.
|
|
||||||
|
|
||||||
Also of record: the plan's "Verification" matrix was
|
|
||||||
executed as written (per-suite gates each stage, the
|
|
||||||
headless datad-feed smoke, the kucoin fail-fast unit
|
|
||||||
check); the known pre-existing ~50% second-runtime-boot
|
|
||||||
test wedge was characterized and excluded as a
|
|
||||||
regression via revert-testing.
|
|
||||||
|
|
@ -1,60 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T21:35:49Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T213549Z_f084e899_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
User bug report from live `ib` testing of the
|
|
||||||
(datad|brokerd)-split branch:
|
|
||||||
|
|
||||||
> ok doing some testing and noticing live orders do not
|
|
||||||
> work otb, pikerd show this on live submission,
|
|
||||||
> [pdb traceback: `RuntimeError("Can not order
|
|
||||||
> {symbol}, no live feed?")` at `ib/api.py:1152`
|
|
||||||
> `submit_limit()` with `self._contracts == {}` in the
|
|
||||||
> `brokerd.ib` actor + a `TrioTaskExited` teardown
|
|
||||||
> cascade]
|
|
||||||
|
|
||||||
Follow-up user direction fixing the design intent:
|
|
||||||
|
|
||||||
> i think the main thing here is that [brokerd] should
|
|
||||||
> be able to always submit orders without a live feed
|
|
||||||
> being up and registered in the same subactor right?
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Root cause: pre-split the feed code populated the
|
|
||||||
actor-local `Client._contracts` cache (via
|
|
||||||
`get_mkt_info()` writing `mkt.bs_fqme` keys) in the
|
|
||||||
SAME process as order submission; post-split the
|
|
||||||
trading actor's client is never warmed. Fix: lazily
|
|
||||||
qualify + cache the contract per order request in
|
|
||||||
`handle_order_requests()` by running the same
|
|
||||||
`get_mkt_info(fqme, proxy=...)` ep the feed side uses,
|
|
||||||
plus per-order error relay (`BrokerdError`) so one bad
|
|
||||||
submission can't crash the whole trades dialog (the
|
|
||||||
`TrioTaskExited` storm was teardown cascade from the
|
|
||||||
original raise).
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/ib/broker.py` — thread `proxies` into
|
|
||||||
`handle_order_requests()`; lazy contract qualify on
|
|
||||||
cache-miss; guard `submit_limit()` w/ `BrokerdError`
|
|
||||||
relay; uncomment the (anticipatory) `get_mkt_info`
|
|
||||||
import
|
|
||||||
- `piker/brokers/ib/api.py` — fix the non-f-string
|
|
||||||
raise msg + document the new qualification contract
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated. Live `ib` order retest
|
|
||||||
performed by the human post-commit.
|
|
||||||
|
|
@ -1,57 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T21:35:49Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); recorded from the live debug
|
|
||||||
session per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/broker.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
|
||||||
|
|
||||||
Key diagnostic chain (from session):
|
|
||||||
|
|
||||||
- pdb showed `Client._contracts == {}` inside
|
|
||||||
`brokerd.ib`'s `submit_limit()`; the cache has
|
|
||||||
exactly TWO write sites: `Client.find_contracts()`
|
|
||||||
(api.py, keys `f'{sym}.{exch}.ib'`) and
|
|
||||||
`symbols.get_mkt_info()` (key `mkt.bs_fqme`, eg.
|
|
||||||
'nvda.nasdaq' — NO `.ib` suffix).
|
|
||||||
- `BrokerdOrder.symbol` arrives as the bs_fqme form
|
|
||||||
('nvda.nasdaq') so ONLY the `get_mkt_info()` write
|
|
||||||
site produces the key `submit_limit()` reads —
|
|
||||||
ie. pre-split it was the feed's in-proc
|
|
||||||
`get_mkt_info(sym, proxy=proxy)` call keeping orders
|
|
||||||
working, NOT `find_contracts()`.
|
|
||||||
- the existing TODO at `symbols.py:642-644` literally
|
|
||||||
predicted this: "this is going to be problematic
|
|
||||||
if/when we split out the datad vs. brokerd actors
|
|
||||||
since the mktmap lookup table will now be
|
|
||||||
inaccessible.."
|
|
||||||
- instance identity verified: `proxy._aio_ns` IS the
|
|
||||||
same `Client` obj as `_accounts2clients[account]`
|
|
||||||
(both sourced from the `load_aio_clients()` cache via
|
|
||||||
`open_client_proxies()`), so a brokerd-side
|
|
||||||
`get_mkt_info(fqme, proxy=proxies[account])` warms
|
|
||||||
exactly the dict `submit_limit()` reads. It also
|
|
||||||
populates `client._cons2mkts` which the
|
|
||||||
position-audit path (`broker.py` backup-table code)
|
|
||||||
needs in this actor anyway.
|
|
||||||
- the `TrioTaskExited` storm in the user's log
|
|
||||||
(`recv_trade_updates`, `open_aio_client_method_relay`
|
|
||||||
aio tasks) is teardown cascade: the raise crashed
|
|
||||||
`handle_order_requests` -> nursery teardown ripped
|
|
||||||
the trio sides of still-running aio relay tasks.
|
|
||||||
Hence the added per-order try/except ->
|
|
||||||
`BrokerdError` relay hardening so a single bad
|
|
||||||
submission degrades to an EMS error msg instead of
|
|
||||||
killing the backend's entire order-ctl dialog.
|
|
||||||
|
|
||||||
Verification: `tests/test_services.py` (5 passed) +
|
|
||||||
`tests/test_ems.py` (6 passed) regression-green; live
|
|
||||||
`ib` submission retest delegated to the human (needs a
|
|
||||||
running TWS/gw).
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
# AI Prompt I/O Log — claude
|
|
||||||
|
|
||||||
This directory tracks prompt inputs and model
|
|
||||||
outputs for AI-assisted development using
|
|
||||||
`claude` (claude-code CLI).
|
|
||||||
|
|
||||||
## Policy
|
|
||||||
|
|
||||||
Prompt logging follows the
|
|
||||||
[NLNet generative AI policy][nlnet-ai].
|
|
||||||
All substantive AI contributions are logged
|
|
||||||
with:
|
|
||||||
- Model name and version
|
|
||||||
- Timestamps
|
|
||||||
- The prompts that produced the output
|
|
||||||
- Unedited model output (`.raw.md` files)
|
|
||||||
|
|
||||||
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
Entries are created by the `/prompt-io` skill
|
|
||||||
or automatically via `/commit-msg` integration.
|
|
||||||
|
|
||||||
Each commit carrying AI-generated changes links
|
|
||||||
to its provenance entry via a `Prompt-IO:`
|
|
||||||
commit-msg trailer; entries use "diff-ref mode"
|
|
||||||
(pointers into `git log -p` instead of verbatim
|
|
||||||
code copies) to avoid duplicating committed
|
|
||||||
code.
|
|
||||||
|
|
||||||
Human contributors remain accountable for all
|
|
||||||
code decisions. AI-generated content is never
|
|
||||||
presented as human-authored work.
|
|
||||||
|
|
@ -1,64 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-21T20:52:28Z
|
|
||||||
git_ref: 58ffe487
|
|
||||||
scope: config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260721T205228Z_58ffe487_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user asked to scan Claude's repo-local skills and
|
|
||||||
documentation, summarize their specializations, and then
|
|
||||||
rework repo-specific skills for use across coding
|
|
||||||
harnesses, particularly OpenCode. The user intends to
|
|
||||||
choose one of Claude's unfinished tasks afterward.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Standardized the tracked piker skills on portable Agent
|
|
||||||
Skills frontmatter, made `commit-msg` harness-aware,
|
|
||||||
documented OpenCode discovery and command delegation, and
|
|
||||||
fixed an unsafe `np.searchsorted()` example found during
|
|
||||||
the audit. Preserved the existing `.claude` artifact path
|
|
||||||
as a shared legacy location to avoid breaking established
|
|
||||||
commit workflows.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `.claude/skills/commit-msg/SKILL.md` - portable,
|
|
||||||
harness-aware commit-message workflow
|
|
||||||
- `.claude/skills/commit-msg/style-guide-reference.md` -
|
|
||||||
generic coding-harness attribution
|
|
||||||
- `.claude/skills/piker-conc-expert/SKILL.md` - portable
|
|
||||||
frontmatter
|
|
||||||
- `.claude/skills/piker-profiling/SKILL.md` - portable
|
|
||||||
frontmatter
|
|
||||||
- `.claude/skills/piker-slang/SKILL.md` - portable
|
|
||||||
frontmatter
|
|
||||||
- `.claude/skills/pyqtgraph-optimization/SKILL.md` -
|
|
||||||
portable frontmatter
|
|
||||||
- `.claude/skills/timeseries-optimization/SKILL.md` -
|
|
||||||
portable frontmatter and safe search bounds handling
|
|
||||||
- `ai/README.md` - shared-skill integration model
|
|
||||||
- `ai/claude-code/README.md` - cross-harness Claude Code
|
|
||||||
documentation
|
|
||||||
- `ai/opencode/README.md` - OpenCode usage and discovery
|
|
||||||
documentation
|
|
||||||
- `.opencode/commands/commit-msg.md` - thin OpenCode
|
|
||||||
command delegating to the shared skill
|
|
||||||
- `ai/prompt-io/opencode/README.md` - OpenCode provenance
|
|
||||||
policy
|
|
||||||
- `ai/prompt-io/opencode/20260721T205228Z_58ffe487_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260721T205228Z_58ffe487_prompt_io.md`
|
|
||||||
- provenance metadata and summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not yet been reviewed or
|
|
||||||
edited by the human.
|
|
||||||
|
|
@ -1,33 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-21T20:52:28Z
|
|
||||||
git_ref: 58ffe487
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
The repo-local skill inventory contains seven domains:
|
|
||||||
commit-message generation, actor-runtime concurrency,
|
|
||||||
distributed profiling, project communication style,
|
|
||||||
Python source style, PyQtGraph rendering, and NumPy/Polars
|
|
||||||
timeseries optimization.
|
|
||||||
|
|
||||||
I converted the six tracked repo-specific skills to the
|
|
||||||
portable Agent Skills frontmatter subset so Claude Code
|
|
||||||
and OpenCode can load the same source files. The
|
|
||||||
commit-message workflow now gathers live git context at
|
|
||||||
execution time, uses runtime harness/model attribution,
|
|
||||||
preserves human-owned task state, and keeps the existing
|
|
||||||
message artifact paths for compatibility.
|
|
||||||
|
|
||||||
I added OpenCode integration documentation and updated
|
|
||||||
the shared and Claude Code docs to describe a single
|
|
||||||
cross-harness skill source with thin harness commands.
|
|
||||||
During validation I also corrected an out-of-bounds
|
|
||||||
`np.searchsorted()` example in the timeseries skill.
|
|
||||||
|
|
||||||
The unfinished artifacts found during the scan are a
|
|
||||||
shared-memory migration plan and a gap-annotation
|
|
||||||
performance handoff. The earlier datad split plan appears
|
|
||||||
implemented and is now captured by the concurrency skill.
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-21T21:50:55Z
|
|
||||||
git_ref: c6ec3d41
|
|
||||||
scope: docs
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260721T215055Z_c6ec3d41_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user asked to boot a plan to continue work on the
|
|
||||||
time-series gap checker and bug fixing of the history
|
|
||||||
backfiller. The user authorized prompt-I/O capture for the
|
|
||||||
plan documentation after the repository audit.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Produced a bounded repair plan based on current source,
|
|
||||||
historical issue caches, existing profiling notes, and a
|
|
||||||
test-coverage audit. The plan prioritizes reproducible
|
|
||||||
correctness failures and storage durability before venue
|
|
||||||
classification, annotation performance, or manual repair
|
|
||||||
UX.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `plans/opencode/gap-checker-backfiller-repair.md` -
|
|
||||||
phased implementation and verification plan
|
|
||||||
- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated plan has not yet been edited by the
|
|
||||||
human.
|
|
||||||
|
|
@ -1,26 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-21T21:50:55Z
|
|
||||||
git_ref: c6ec3d41
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff -- plans/opencode/gap-checker-backfiller-repair.md`
|
|
||||||
|
|
||||||
Generated a phased implementation plan for continuing
|
|
||||||
piker's gap checker and history backfiller repairs. The
|
|
||||||
plan separates data correctness from annotation
|
|
||||||
performance and orders work by dependency: deterministic
|
|
||||||
tests, canonical frame/gap normalization, structured
|
|
||||||
backfill completion, non-destructive atomic NativeDB
|
|
||||||
writes, exact null repair, provider/venue classification,
|
|
||||||
annotation vectorization, and crash qualification.
|
|
||||||
|
|
||||||
The plan is grounded in current code paths, local issue
|
|
||||||
caches `#62`, `#71`, and `#75`, and the existing 1,285-gap
|
|
||||||
profiling handoff. It records confirmed present defects,
|
|
||||||
design invariants, per-phase acceptance gates, test
|
|
||||||
commands, commit boundaries, and explicitly deferred UX
|
|
||||||
and provider work.
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-22T00:13:57Z
|
|
||||||
git_ref: 16f3cd6f
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260722T001357Z_16f3cd6f_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Continue the gap-checker and history-backfiller repair plan. Start
|
|
||||||
with deterministic analysis characterization, then fix canonical
|
|
||||||
sorting, dedupe-derived columns, positive gap detection, and null
|
|
||||||
grouping.
|
|
||||||
|
|
||||||
During review, the user challenged the replacement null-grouping
|
|
||||||
algorithm because its initial tests did not assert exact endpoints.
|
|
||||||
The user explicitly required retaining detailed, human-oriented
|
|
||||||
algorithm documentation such as the existing ASCII segment map.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Added deterministic analysis regressions and corrected normalization
|
|
||||||
so sorting and duplicate resolution precede derived-column
|
|
||||||
calculation. Unified null grouping for NumPy and Polars inputs,
|
|
||||||
documented its exact inclusive-boundary contract with an adapted
|
|
||||||
visual map, and verified all small null layouts against an independent
|
|
||||||
state-machine oracle. Retained sorted smart-dedupe output.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/tsp/_anal.py` - canonical gap and documented null grouping
|
|
||||||
- `piker/tsp/_dedupe_smart.py` - sorted duplicate resolution
|
|
||||||
- `tests/test_tsp_analysis.py` - exact and exhaustive regressions
|
|
||||||
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not yet been edited by the human.
|
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-22T00:13:57Z
|
|
||||||
git_ref: 16f3cd6f
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
|
|
||||||
|
|
||||||
Generated canonical gap and null-segment normalization. Derived
|
|
||||||
datetime and delta columns are recomputed after final sorting and
|
|
||||||
duplicate resolution, and negative deltas no longer become ordinary
|
|
||||||
gaps.
|
|
||||||
|
|
||||||
NumPy and Polars null grouping share one implementation while
|
|
||||||
retaining the human-oriented visual map of contiguous zero runs. The
|
|
||||||
documented contract defines absolute inclusive endpoints, margin
|
|
||||||
expansion, frame boundary clamping, contiguous-index validation, and
|
|
||||||
requested-column selection.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_dedupe_smart.py`
|
|
||||||
|
|
||||||
Adjusted smart dedupe to honor sorted output and resolve duplicate
|
|
||||||
timestamps without retaining stale order.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
|
|
||||||
|
|
||||||
Added deterministic synthetic OHLCV coverage for sorted dedupe, fresh
|
|
||||||
derived columns, and positive gap detection. Null grouping is checked
|
|
||||||
against readable endpoint examples and an independent state-machine
|
|
||||||
oracle over every layout through six rows, three margin widths, and
|
|
||||||
both NumPy and Polars inputs.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
`timeout -k 5 30 python -m pytest -q tests/test_tsp_analysis.py`
|
|
||||||
|
|
||||||
Result: 11 passed.
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-22T00:13:58Z
|
|
||||||
git_ref: 16f3cd6f
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260722T001358Z_16f3cd6f_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Continue the structured backfill phase after the analysis regressions.
|
|
||||||
Characterize provider exhaustion and genuinely empty frames, then remove
|
|
||||||
completion paths that can hang after child-task exit.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Removed orphan-prone completion events from the structured nursery flow
|
|
||||||
and checked empty provider frames before indexing timestamp endpoints.
|
|
||||||
Added deterministic timeout-bounded regressions for both exit paths.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/tsp/_history.py` - structured completion and empty-frame fixes
|
|
||||||
- `tests/test_history_backfill.py` - deterministic exit regressions
|
|
||||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not yet been edited by the human.
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-22T00:13:58Z
|
|
||||||
git_ref: 16f3cd6f
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
|
||||||
|
|
||||||
Removed redundant completion events whose consumers waited only after
|
|
||||||
their owning nurseries had joined. Added empty provider-frame handling
|
|
||||||
before timestamp endpoint indexing so provider exhaustion exits cleanly.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
|
||||||
|
|
||||||
Added bounded, actor-free regressions for `DataUnavailable` and empty
|
|
||||||
provider-frame completion.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
`timeout -k 5 60 python -m pytest -q tests/test_history_backfill.py`
|
|
||||||
|
|
||||||
Result: 2 passed.
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-22T02:41:33Z
|
|
||||||
git_ref: 3a4f6737
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260722T024133Z_3a4f6737_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user challenged the analysis commit message's claim that only
|
|
||||||
positive sample-step gaps are detected and asked whether out-of-order
|
|
||||||
timestamps were simply ignored. After confirming the missing separate
|
|
||||||
reporting path, the user instructed the agent to implement the fix.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Separated missing-history gaps from timestamp-ordering errors without
|
|
||||||
changing the existing `dedupe()` return shape. Added a structured
|
|
||||||
detector and made normalization warn before sorting removes evidence of
|
|
||||||
duplicates or reversed timestamps.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/tsp/_anal.py` - ordering detector and dedupe reporting
|
|
||||||
- `piker/tsp/__init__.py` - public detector export
|
|
||||||
- `tests/test_tsp_analysis.py` - exact ordering-error regression
|
|
||||||
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not yet been edited by the human.
|
|
||||||
|
|
@ -1,30 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-22T02:41:33Z
|
|
||||||
git_ref: 3a4f6737
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
|
|
||||||
|
|
||||||
Added `detect_time_ordering_errors()` to return non-positive timestamp
|
|
||||||
steps as structured rows. `dedupe()` now warns with those rows before
|
|
||||||
sorting and duplicate removal erase the original ordering evidence.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/__init__.py`
|
|
||||||
|
|
||||||
Exported the ordering detector through the public TSP package.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
|
|
||||||
|
|
||||||
Changed the negative-delta regression to verify exact predecessor,
|
|
||||||
timestamp, and delta values, confirm exclusion from missing-history
|
|
||||||
gaps, and require explicit normalization reporting.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- `tests/test_tsp_analysis.py`: 11 passed
|
|
||||||
- Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-22T02:41:34Z
|
|
||||||
git_ref: 3a4f6737
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260722T024134Z_3a4f6737_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user asked why the in-progress `piker/storage` changes were omitted
|
|
||||||
from the earlier commit plan. After receiving an inventory of unfinished
|
|
||||||
Phase 3 requirements, the user instructed the agent to complete the work.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Finished non-destructive, atomic NativeDB updates and integrated them
|
|
||||||
with history startup and reverse backfill. Expanded tests through two
|
|
||||||
adversarial review rounds that caught and corrected cross-client lock
|
|
||||||
blocking, stale temp indexing, discontinuous persisted indexes, startup
|
|
||||||
snapshot races, missing latest-frame persistence, notification wedges,
|
|
||||||
and mutable/truncated SHM persistence.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/storage/__init__.py` - incremental storage contract
|
|
||||||
- `piker/storage/nativedb.py` - validated atomic merge persistence
|
|
||||||
- `piker/storage/marketstore/__init__.py` - non-duplicate update mode
|
|
||||||
- `piker/tsp/_history.py` - provider-delta and startup persistence
|
|
||||||
- `tests/test_storage_nativedb.py` - NativeDB durability regressions
|
|
||||||
- `tests/test_history_backfill.py` - persistence lifecycle regressions
|
|
||||||
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not yet been edited by the human.
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-22T02:41:34Z
|
|
||||||
git_ref: 3a4f6737
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/storage/__init__.py piker/storage/nativedb.py piker/storage/marketstore/__init__.py`
|
|
||||||
|
|
||||||
Added an incremental storage update contract. NativeDB now validates
|
|
||||||
canonical numeric OHLCV data, serializes writers both actor-locally and
|
|
||||||
through cancellable filesystem locks, merges with incoming-wins seam
|
|
||||||
resolution, regenerates contiguous indexes, reopens and validates a
|
|
||||||
same-directory temporary parquet, fsyncs it, atomically replaces the
|
|
||||||
target, fsyncs the directory, and publishes cache only after replacement.
|
|
||||||
|
|
||||||
NativeDB indexing ignores lock and crash-left temporary files.
|
|
||||||
MarketStore uses its non-duplicate write mode for incremental updates.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
|
||||||
|
|
||||||
Separated full provider deltas from capacity-truncated SHM pushes.
|
|
||||||
Backfill persists before SHM mutation, publishes bounded notifications,
|
|
||||||
and delays latest-frame publication until the concurrent pre-update
|
|
||||||
storage snapshot completes.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py tests/test_history_backfill.py`
|
|
||||||
|
|
||||||
Added deterministic coverage for merge preservation, conflict policy,
|
|
||||||
atomic failures, reopened-candidate validation, cache consistency,
|
|
||||||
schema and timestamp invariants, contiguous index regeneration,
|
|
||||||
separate series, stale temp indexing, cancellable file-lock contention,
|
|
||||||
full provider-delta persistence, latest-frame ordering, and bounded
|
|
||||||
notification teardown.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- deterministic analysis/backfill/storage set: 31 passed
|
|
||||||
- Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
- final adversarial review: no findings
|
|
||||||
|
|
@ -1,32 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: ses_092a25886ffeT0nrIrdkMEcgJK
|
|
||||||
timestamp: 2026-07-24T22:01:56Z
|
|
||||||
git_ref: flake_update
|
|
||||||
scope: config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260724T220156Z_43d7a3ed_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
After committing Piker's provider-neutral deployment, the user asked to
|
|
||||||
extract Piker-specialized `/run-tests` content from the historical
|
|
||||||
`wkt/run_tests_skill` implementation into the new deployment.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Create a repository-owned `test-harness-reference.md` for the canonical Run
|
|
||||||
base. Revalidate historical commands and assumptions against current pytest,
|
|
||||||
uv/Nix environments, tests, fixtures, and Tractor runtime behavior, retaining
|
|
||||||
only current Piker-specific guidance.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `.claude/skills/run-tests/test-harness-reference.md` - define Piker's safe
|
|
||||||
commands, opt-in boundaries, mappings, known outcomes, and runtime notes.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None at generation time; pending human review.
|
|
||||||
|
|
@ -1,32 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-24T22:01:56Z
|
|
||||||
git_ref: flake_update
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
The user asked to extract useful Piker-specific `/run-tests` behavior from
|
|
||||||
`wkt/run_tests_skill` after committing the provider-neutral deployment.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
|
|
||||||
Generate a repository-owned harness reference rather than restoring the old
|
|
||||||
standalone `SKILL.md`. Preserve verified Piker-specific environment, command,
|
|
||||||
scope, fixture, test-layout, change-mapping, known-outcome, and Tractor-runtime
|
|
||||||
details while leaving shared execution and cleanup policy in the canonical
|
|
||||||
`ai.skillz` base.
|
|
||||||
|
|
||||||
Repository inspection found that the historical skill needs correction:
|
|
||||||
|
|
||||||
- the comments-only `pytest.ini` shadows `pyproject.toml` pytest options, so
|
|
||||||
`-p no:xonsh` must be explicit;
|
|
||||||
- no marker-based offline suite exists, and several live/GUI/container tests
|
|
||||||
are unsafe defaults;
|
|
||||||
- skipped Docker and Questrade modules can fail during import before marks;
|
|
||||||
- actor timeout retry remains relevant only for the exact timed-out command;
|
|
||||||
- `py313` and frozen no-sync `uv run` are the current local environment paths.
|
|
||||||
|
|
||||||
Validate with checkout import resolution, safe collection, and deterministic
|
|
||||||
watchlist/accounting tests. Do not run live broker, GUI, container,
|
|
||||||
credentialed, or user-config-dependent cases during harness verification.
|
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: unavailable
|
|
||||||
timestamp: 2026-07-27T20:47:11Z
|
|
||||||
git_ref: 84a6d47b
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260727T204711Z_84a6d47b_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user identified that `backfiller_deep_fixes` introduced persistent
|
|
||||||
`.parquet.lock` files, questioned whether Parquet or Arrow already handled
|
|
||||||
writer coordination, and clarified that piker's `tractor` structured
|
|
||||||
concurrency design already provides single-writer ownership through the
|
|
||||||
persistent `datad` feed task. The user then asked to open the backfiller
|
|
||||||
worktree and resolve the redundant locking.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Removed redundant actor-local and filesystem locking while preserving
|
|
||||||
atomic crash-safe Parquet replacement. Replaced the synthetic contention
|
|
||||||
test with regressions for the actual cross-branch failure: writes create
|
|
||||||
no lock sidecars, and legacy lock artifacts are ignored during indexing.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/storage/nativedb.py` - rely on `datad` writer ownership
|
|
||||||
- `tests/test_storage_nativedb.py` - lock-sidecar regressions
|
|
||||||
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,30 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-27T20:47:11Z
|
|
||||||
git_ref: 84a6d47b
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py`
|
|
||||||
|
|
||||||
Removed actor-local and filesystem writer locks from NativeDB. The
|
|
||||||
persistent `datad` feed task already owns writes for each series, while
|
|
||||||
its timeframe children write distinct files. NativeDB retains validated
|
|
||||||
temporary writes, file and directory synchronization, and atomic target
|
|
||||||
replacement for crash durability. The exact `.parquet` suffix filter is
|
|
||||||
retained so legacy lock sidecars are harmless.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py`
|
|
||||||
|
|
||||||
Removed the synthetic cross-client lock-contention test. Added regression
|
|
||||||
coverage proving replacement and incremental writes create no lock
|
|
||||||
sidecars. Extended indexing coverage with the legacy
|
|
||||||
`.parquet.lock` artifact that caused upstream `flake_update` to crash.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- NativeDB and history regression set: 20 passed
|
|
||||||
- Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
|
|
@ -1,42 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-27T21:24:54Z
|
|
||||||
git_ref: 689df816
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260727T212454Z_689df816_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user reran the known IB FQME on `backfiller_deep_fixes` and reported
|
|
||||||
that `publish_latest_frame()` still failed because NativeDB required an
|
|
||||||
`index` absent from IB's provider frame. The user called out the missed
|
|
||||||
tabular schema boundary and requested an actual end-to-end test suite
|
|
||||||
rather than storage fakes.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Repaired the IB-provider-to-NativeDB boundary by canonicalizing durable
|
|
||||||
fields and mapping reloads into provider-specific SHM buffers. Added a
|
|
||||||
local actor/SHM/Parquet integration regression covering first startup and
|
|
||||||
restart, plus focused conversion and timestamp-coercion tests. Two review
|
|
||||||
rounds found and resolved restart hydration and fractional-time hazards.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/storage/nativedb.py` - canonical storage and SHM field mapping
|
|
||||||
- `piker/tsp/_anal.py` - name-based Polars-to-NumPy conversion
|
|
||||||
- `tests/test_history_backfill.py` - actor/SHM/Parquet integration test
|
|
||||||
- `tests/test_storage_nativedb.py` - schema conversion regressions
|
|
||||||
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-27T21:24:54Z
|
|
||||||
git_ref: 689df816
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py piker/tsp/_anal.py`
|
|
||||||
|
|
||||||
Normalized index-less provider frames to NativeDB's canonical names,
|
|
||||||
order, and dtypes before merge and publication. Provider-only columns
|
|
||||||
are excluded from durable files, derived indexes are regenerated, and
|
|
||||||
fractional timestamps are rejected before integer conversion. Added a
|
|
||||||
canonical field map for hydrating provider-specific SHM buffers while
|
|
||||||
leaving provider-only fields at their defaults. Changed Polars-to-NumPy
|
|
||||||
conversion to select columns by field name rather than position.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py tests/test_storage_nativedb.py`
|
|
||||||
|
|
||||||
Added deterministic coverage using IB's actual provider dtype and a real
|
|
||||||
local `tractor` root actor, IB-typed shared-memory buffers, NativeDB,
|
|
||||||
Parquet persistence, first-start publication, and restart hydration.
|
|
||||||
Added focused regressions for reordered/extra Polars columns and
|
|
||||||
fractional timestamp rejection.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- NativeDB and history integration set: 23 passed
|
|
||||||
- Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
- second adversarial review: no findings
|
|
||||||
- no live gateway or network tests run
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-27T21:59:06Z
|
|
||||||
git_ref: 689df816
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260727T215906Z_689df816_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
After the actor/SHM/Parquet integration test exposed a stale SHM triplet,
|
|
||||||
the user requested leak-cleaner machinery in the pytest harness using
|
|
||||||
Tractor's existing ownership and lifetime patterns.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Added creator-scoped SHM leak tracking to pytest without scanning
|
|
||||||
`/dev/shm` or treating attachments as owned. The fixture cooperates with
|
|
||||||
normal Tractor teardown, verifies POSIX object identity before fallback
|
|
||||||
cleanup, restores process-local token state, and fails after cleaning a
|
|
||||||
leak. Regressions cover the exact pre-actor-registration allocation window,
|
|
||||||
positional creators, and external attachment safety.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `tests/conftest.py` - creator-scoped SHM leak tracking fixture
|
|
||||||
- `tests/test_shm_cleanup.py` - ownership and failure-window regressions
|
|
||||||
- `.claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
- document current-process SHM cleanup behavior
|
|
||||||
- `ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-27T21:59:06Z
|
|
||||||
git_ref: 689df816
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/conftest.py tests/test_shm_cleanup.py .claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
|
|
||||||
Added function-scoped SHM ownership tracking around Tractor's test-process
|
|
||||||
factory. The fixture records only successful creators, removes ownership
|
|
||||||
records during normal Tractor teardown, verifies POSIX object identity
|
|
||||||
before fallback unlink, restores the token-cache baseline, and fails after
|
|
||||||
cleaning any leak. Regressions reproduce allocation failure before actor
|
|
||||||
lifetime registration and prove external attachments remain untouched.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- SHM cleanup regressions: 2 passed
|
|
||||||
- combined SHM, NativeDB, and history set: 25 passed
|
|
||||||
- Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
- final adversarial review: no findings
|
|
||||||
|
|
@ -1,42 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
|
||||||
timestamp: 2026-07-28T00:32:47Z
|
|
||||||
git_ref: ed85721c
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260728T003247Z_ed85721c_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Build a safe xonsh workflow for qualifying NativeDB and reverse-backfill
|
|
||||||
correctness against a known gappy IB chart, beginning with
|
|
||||||
`mnq.cme.20260918`. Cover baseline, fresh, restart, append, and expansion
|
|
||||||
testing through implementation, adversarial review, and verification.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Implemented a private disposable xonsh qualification workflow. Evidence is
|
|
||||||
tied to a fixed case, report checksum, snapshot checksum, worktree import,
|
|
||||||
and phase metadata. Added explicit path and symlink safety, source
|
|
||||||
replacement handling, xonsh child environments, and lifecycle preservation
|
|
||||||
comparisons.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `snippets/nativedb_backfill_audit.xsh` - disposable qualification helpers
|
|
||||||
- `docs/manual_backfill_qualification.rst` - operator lifecycle runbook
|
|
||||||
- `tests/test_backfill_audit_snippet.py` - xonsh and comparison regressions
|
|
||||||
- `.claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
- qualification test-harness guidance
|
|
||||||
- `ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,32 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-28T00:32:47Z
|
|
||||||
git_ref: ed85721c
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- snippets/nativedb_backfill_audit.xsh docs/manual_backfill_qualification.rst tests/test_backfill_audit_snippet.py .claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
|
|
||||||
Added sourceable xonsh helpers that create a private disposable root, bind
|
|
||||||
child processes to the reviewed worktree, capture checksum-bound phase
|
|
||||||
evidence, safely seed and clear test history, and compare lifecycle
|
|
||||||
preservation.
|
|
||||||
|
|
||||||
The workflow rejects path traversal, pre-existing symlinks, output
|
|
||||||
collisions, mismatched case/report/snapshot evidence, and same-phase
|
|
||||||
comparison. The accompanying runbook covers known-bad replay, fresh
|
|
||||||
backfill, restart, append, gap triage, and expansion cases.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- xonsh qualification helper regressions: passed
|
|
||||||
- xonsh sourceability: passed
|
|
||||||
- Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
|
|
||||||
Adversarial review drove additional evidence binding, worktree executable,
|
|
||||||
environment propagation, source replacement, and descriptor-level safety
|
|
||||||
fixes. Residual concurrent mutation risk is stated explicitly in the private
|
|
||||||
disposable-root runbook.
|
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
|
||||||
timestamp: 2026-07-28T00:50:34Z
|
|
||||||
git_ref: ed85721c
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260728T005034Z_ed85721c_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Add reusable read-only tooling to qualify raw NativeDB Parquet against a
|
|
||||||
known gappy IB chart, beginning with `mnq.cme.20260918`. Preserve malformed
|
|
||||||
evidence and distinguish structural validity from unclassified positive
|
|
||||||
gaps without opening or mutating the storage runtime.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Implemented raw Parquet audit reporting with human and JSON output, bounded
|
|
||||||
cadence details, exact integer arithmetic, strict status, collision-safe
|
|
||||||
snapshot capture, and source path protections. Regressions cover malformed
|
|
||||||
schema and values, timestamp precision, gap arithmetic, corrupt bytes,
|
|
||||||
copy failures, symlinks, traversal, and read-only CLI behavior.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/storage/_audit.py` - raw Parquet audit and snapshot implementation
|
|
||||||
- `piker/storage/cli.py` - `piker store audit` command
|
|
||||||
- `tests/test_storage_audit.py` - audit and CLI regressions
|
|
||||||
- `ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-28T00:50:34Z
|
|
||||||
git_ref: ed85721c
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/storage/_audit.py piker/storage/cli.py tests/test_storage_audit.py`
|
|
||||||
|
|
||||||
Added a raw, read-only NativeDB Parquet audit with human and JSON output,
|
|
||||||
bounded cadence evidence, exact integer arithmetic, pre-parse snapshot
|
|
||||||
capture, and strict qualification status.
|
|
||||||
|
|
||||||
The command rejects unsafe FQME paths, symlinked sources and outputs,
|
|
||||||
destination collisions, non-regular source nodes, malformed canonical
|
|
||||||
schema, non-finite values, invalid timestamps, noncanonical indexes, and
|
|
||||||
sub-period cadence. Copy-failure cleanup preserves source bytes while
|
|
||||||
completed corrupt snapshots remain available for diagnosis.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- storage audit and CLI regressions: passed
|
|
||||||
- Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
- final adversarial review: no findings
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
|
||||||
timestamp: 2026-07-28T23:27:18Z
|
|
||||||
git_ref: ed85721c
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260728T232718Z_ed85721c_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Diagnose the live `pikerd` failure where backfill publication raised
|
|
||||||
`ValueError: OHLCV timestamps must be finite and positive` while merging the
|
|
||||||
known gappy MNQ NativeDB baseline with fresh IB history. Determine whether to
|
|
||||||
commit the pending audit stack or iterate first.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Traced the primary failure through `publish_latest_frame()` to
|
|
||||||
`NativeStorageClient.update_ohlcv()`. The incoming frame was valid, but the
|
|
||||||
canonicalized persisted baseline retained its epoch-zero row and caused the
|
|
||||||
merged frame to fail validation. Added persisted-row repair before merge and
|
|
||||||
a regression using the observed legacy schema characteristics.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/storage/nativedb.py` - drop non-positive persisted rows during merge
|
|
||||||
- `tests/test_storage_nativedb.py` - live MNQ failure-shape regression
|
|
||||||
- `ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-28T23:27:18Z
|
|
||||||
git_ref: ed85721c
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py tests/test_storage_nativedb.py`
|
|
||||||
|
|
||||||
Fixed incremental NativeDB updates against legacy persisted history with
|
|
||||||
non-positive timestamps. Incoming provider frames remain strictly validated,
|
|
||||||
while merge repair discards invalid persisted rows before concat, dedupe,
|
|
||||||
ordering, canonical-index rewrite, and atomic publication.
|
|
||||||
|
|
||||||
The regression reproduces the observed MNQ baseline shape: one epoch-zero
|
|
||||||
row, noncanonical absolute indexes, extra derived columns, and a valid fresh
|
|
||||||
IB frame. It proves the invalid row is removed while all valid old and new
|
|
||||||
bars survive with canonical indexes.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- NativeDB regressions: 18 passed
|
|
||||||
- history backfill regressions: 6 passed
|
|
||||||
- focused Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
|
||||||
timestamp: 2026-07-29T03:41:06Z
|
|
||||||
git_ref: 922a5df8
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260729T034106Z_922a5df8_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Diagnose the live IB reverse-fill failure where
|
|
||||||
`reqHistoricalDataAsync()` raised API error 10314 because its end date,
|
|
||||||
time, or timezone was invalid.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Traced the failure to `Client.bars()`, where an unfinished EST conversion
|
|
||||||
left the original datetime for `ib_async` to serialize as
|
|
||||||
`YYYYMMDD HH:MM:SS UTC`. Converted aware boundaries explicitly to UTC and
|
|
||||||
passed IB's accepted `YYYYMMDD-HH:MM:SS` string while retaining a blank
|
|
||||||
latest-request boundary. Added exact request-argument regressions.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/ib/api.py` - explicit UTC-dash history boundary format
|
|
||||||
- `tests/test_ib_history.py` - IB request-format regressions
|
|
||||||
- `.claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
- deterministic IB history test mapping
|
|
||||||
- `ai/prompt-io/opencode/20260729T034106Z_922a5df8_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260729T034106Z_922a5df8_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-29T03:41:06Z
|
|
||||||
git_ref: 922a5df8
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py tests/test_ib_history.py .claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
|
|
||||||
Fixed IB reverse-history requests to serialize aware end boundaries with
|
|
||||||
IB's explicit UTC-dash syntax, `YYYYMMDD-HH:MM:SS`. This bypasses
|
|
||||||
`ib_async`'s trailing-`UTC` representation, which the live Gateway rejected
|
|
||||||
with API error 10314, and removes the unfinished EST comparison breakpoint.
|
|
||||||
|
|
||||||
The regression captures the exact `reqHistoricalDataAsync()` argument for
|
|
||||||
latest, stdlib-aware, Pendulum UTC, and America/New_York inputs.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- IB history request regressions: 4 passed
|
|
||||||
- focused Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
- adversarial review: no findings
|
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
|
||||||
timestamp: 2026-07-29T04:17:23Z
|
|
||||||
git_ref: ce33deb6
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260729T041723Z_ce33deb6_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Diagnose why the chart still showed a zero first OHLCV row after the
|
|
||||||
epoch-zero timestamp repair, causing auto-y-ranging to include zero.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Audited live 60s and 1s NativeDB files and found one exact 60s first-slot
|
|
||||||
sentinel at epoch 60 with a completely zero OHLCV payload. Added narrowly
|
|
||||||
scoped read and merge repair, matching incoming validation, sentinel-only
|
|
||||||
startup handling, canonical reindexing, and explicit audit classification.
|
|
||||||
Preserved modern all-zero bars as unclassified evidence rather than assuming
|
|
||||||
all zero-priced instruments are corrupt.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/storage/nativedb.py` - first-slot sentinel repair and validation
|
|
||||||
- `piker/storage/_audit.py` - sentinel classification and zero-row evidence
|
|
||||||
- `tests/test_storage_nativedb.py` - hydration, merge, and boundary regressions
|
|
||||||
- `tests/test_storage_audit.py` - sentinel and modern-zero audit regressions
|
|
||||||
- `ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not been edited by the human.
|
|
||||||
|
|
@ -1,31 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-29T04:17:23Z
|
|
||||||
git_ref: ce33deb6
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py piker/storage/_audit.py tests/test_storage_nativedb.py tests/test_storage_audit.py`
|
|
||||||
|
|
||||||
Fixed the remaining legacy first-slot sentinel observed in live MNQ 60s
|
|
||||||
history: `time=60` with zero open, high, low, close, and volume. NativeDB now
|
|
||||||
repairs the exact sentinel before read hydration and merge, reindexes the
|
|
||||||
filtered view, and persists the repaired frame on the next valid update.
|
|
||||||
|
|
||||||
The repair remains narrowly scoped. Modern all-zero OHLC bars stay valid for
|
|
||||||
spreads or synthetic instruments, null-containing malformed rows still fail
|
|
||||||
validation, new writes can not recreate the first-slot sentinel, and a file
|
|
||||||
containing only sentinels loads as no history so fresh backfill can proceed.
|
|
||||||
|
|
||||||
The raw audit reports total all-zero price rows, classifies first-slot
|
|
||||||
sentinels as structural violations, and leaves later all-zero rows visible as
|
|
||||||
unclassified warnings.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
- NativeDB, audit, history, xonsh, and IB regressions: 61 passed
|
|
||||||
- focused Ruff: passed
|
|
||||||
- `git diff --check`: passed
|
|
||||||
- final adversarial review: no findings
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: 6409e3bb-1a9a-4ada-b6c4-c554115143e9
|
|
||||||
timestamp: 2026-07-29T18:54:06Z
|
|
||||||
git_ref: d359b1cb
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260729T185406Z_d359b1cb_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user reported that QQQ 60-second zeros appeared as a flat line after
|
|
||||||
reload, clarified that the outstanding concern was the prior `ldshm`
|
|
||||||
failure, and instructed the agent to implement the more focused fix.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Changed `ldshm` to classify invalid SHM snapshots before downstream
|
|
||||||
analysis. Cadence is inferred only from observed finite positive steps;
|
|
||||||
buffers containing non-finite or non-positive timestamps, or lacking a
|
|
||||||
positive cadence, are skipped rather than partially replacement-written
|
|
||||||
or reloaded. Added deterministic helper and command-level regressions.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/storage/cli.py` - inspect cadence and reject invalid SHM snapshots before mutation.
|
|
||||||
- `tests/test_ldshm.py` - cover QQQ-like zeros, corrupt timestamps, cadence selection, degenerate buffers, and mutation prevention.
|
|
||||||
- `.claude/skills/run-tests/test-harness-reference.md` - register the deterministic regression target.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated output remains uncommitted.
|
|
||||||
|
|
@ -1,58 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-29T18:54:06Z
|
|
||||||
git_ref: d359b1cb
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
The user clarified that the QQQ 60-second chart now displayed a flat
|
|
||||||
forward-filled segment, then asked whether the previously reported
|
|
||||||
`piker store ldshm` failure had been fixed. After being told it had not,
|
|
||||||
the user instructed: "ok but i read above you have a more focussed fix
|
|
||||||
todo now? so have at it!"
|
|
||||||
|
|
||||||
The implementation focused on the exact failure boundary. It does not
|
|
||||||
filter invalid rows into a partial replacement and does not mutate live
|
|
||||||
SHM. Instead, `ldshm` inspects one captured Polars snapshot, infers its
|
|
||||||
cadence from observed finite positive timestamp steps, and skips the
|
|
||||||
entire buffer before analysis, persistence, or reload when any timestamp
|
|
||||||
is non-finite or non-positive.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/storage/cli.py`
|
|
||||||
|
|
||||||
Generated `_shm_period_and_invalid_count()` and integrated its result in
|
|
||||||
`ldshm`. The helper selects the most frequent observed positive step,
|
|
||||||
using the smallest observed step as the deterministic tie-break. Short
|
|
||||||
or cadence-degenerate buffers are skipped. Invalid snapshots are logged
|
|
||||||
and cannot reach deduplication, NativeDB replacement, or SHM reload.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
|
|
||||||
|
|
||||||
Generated deterministic regressions for the QQQ-shaped interior zero
|
|
||||||
run, observed-step cadence selection, all invalid timestamp classes,
|
|
||||||
short and degenerate buffers, and command-level prevention of dedupe,
|
|
||||||
storage writes, and reload. The command-level test uses an immutable
|
|
||||||
Polars snapshot and failing spies, without live SHM mutation or timing.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
|
|
||||||
Added the deterministic `ldshm` regression target and its change-to-test
|
|
||||||
mapping to the repository test harness reference.
|
|
||||||
|
|
||||||
Verification output:
|
|
||||||
|
|
||||||
```text
|
|
||||||
.......................................................... [100%]
|
|
||||||
58 passed in 0.70s
|
|
||||||
```
|
|
||||||
|
|
||||||
`git diff --check` passed. Ruff was unavailable in the worktree's
|
|
||||||
existing environment. Three adversarial review passes found and drove
|
|
||||||
fixes for partial-replacement data loss, invented median timeframes,
|
|
||||||
mixed SHM observations, and pre-guard null-analysis failures. The final
|
|
||||||
review reported no remaining findings.
|
|
||||||
|
|
||||||
The patch intentionally does not claim that `ldshm` is safe to run as a
|
|
||||||
general concurrent repair writer. It handles invalid live snapshots by
|
|
||||||
refusing mutation rather than weakening NativeDB validation.
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: 253ade33-07ba-45b5-8839-be35d17ec164
|
|
||||||
timestamp: 2026-07-29T20:40:46Z
|
|
||||||
git_ref: ad6c560f
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260729T204046Z_ad6c560f_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user reported that the QQQ 1-second chart still contained a gap after
|
|
||||||
the `ldshm` fix, asked the agent to inspect it, and authorized continued
|
|
||||||
implementation after a short pause and shutdown of the live workspace.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Diagnosed a 653-row active RT SHM null reservation caused by inclusive IB
|
|
||||||
reverse-query overlaps and startup repair orchestration. Prevented extra
|
|
||||||
physical SHM overlap rows, serialized and bounded IB history recovery,
|
|
||||||
made blank/error responses terminate, and constrained synthetic repair to
|
|
||||||
valid interior timestamp boundaries. Added deterministic concurrency and
|
|
||||||
data-layout regressions.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/ib/feed.py` - bound and serialize IB history reset/retry behavior.
|
|
||||||
- `piker/tsp/_history.py` - prevent SHM overlap skew and safely finalize null repair.
|
|
||||||
- `tests/test_ib_history.py` - cover blank, cancellation, reset, and pacing interleavings.
|
|
||||||
- `tests/test_history_backfill.py` - cover overlap and null-repair invariants.
|
|
||||||
- `.claude/skills/run-tests/test-harness-reference.md` - register deterministic coverage.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated output remains uncommitted.
|
|
||||||
|
|
@ -1,72 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-29T20:40:46Z
|
|
||||||
git_ref: ad6c560f
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
The user reported that QQQ still displayed a gap on the 1-second chart
|
|
||||||
after the focused `ldshm` fix and asked the agent to inspect it. The user
|
|
||||||
then paused and resumed implementation while shutting down the live piker
|
|
||||||
trading workspace.
|
|
||||||
|
|
||||||
Read-only live inspection found the active datad generation held 653
|
|
||||||
contiguous zero-time rows in QQQ's 1-second RT SHM, while durable Parquet
|
|
||||||
contained no zero rows. The run occupied absolute indexes 168148 through
|
|
||||||
168800 between timestamps 1785348398 and 1785349051. The stale actor
|
|
||||||
generation and active 60-second history were separately identified.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
|
||||||
|
|
||||||
Generated history changes that exclude each inclusive reverse-query
|
|
||||||
endpoint already present in SHM while retaining the full provider frame
|
|
||||||
for NativeDB merge. Startup null repair now runs after reverse backfill,
|
|
||||||
uses bounded UI notifications, handles provider no-data without an
|
|
||||||
interactive pause, fills only exact interior zero groups, and refuses
|
|
||||||
synthetic timestamps unless both valid boundaries establish the expected
|
|
||||||
cadence.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
|
|
||||||
|
|
||||||
Generated IB history reset changes that make blank responses complete,
|
|
||||||
bound repeated cancellation and reset failures per request, serialize
|
|
||||||
timeout and pacing farm resets with one actor-global Trio lock, carry
|
|
||||||
explicit reset results, and coordinate pacing handoff with queued timeout
|
|
||||||
waiters.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
|
||||||
|
|
||||||
Generated regressions for inclusive endpoint exclusion, full NativeDB
|
|
||||||
provider deltas, strict synthetic gap boundaries, empty-provider local
|
|
||||||
fallback, bounded notifications, and preservation of valid margin rows.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_ib_history.py`
|
|
||||||
|
|
||||||
Generated deterministic regressions for blank IB responses, cancellation
|
|
||||||
limits, stalled reset cancellation, pacing reset failure limits,
|
|
||||||
cross-request reset serialization, timeout-to-pacing handoff, and
|
|
||||||
successful handoff timeout restart.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
|
|
||||||
Registered the deterministic history-backfill target and expanded the IB
|
|
||||||
history mapping to include `feed.py`.
|
|
||||||
|
|
||||||
Verification output:
|
|
||||||
|
|
||||||
```text
|
|
||||||
........................................................................ [ 93%]
|
|
||||||
..... [100%]
|
|
||||||
77 passed in 2.12s
|
|
||||||
```
|
|
||||||
|
|
||||||
`git diff --check` passed. Multiple adversarial review rounds identified
|
|
||||||
and drove fixes for unbounded IB empty/cancellation/reset paths, stale
|
|
||||||
reset ownership, actor-global failure poisoning, concurrent farm resets,
|
|
||||||
double-counted pacing handoffs, physical endpoint overlap, and unsafe
|
|
||||||
synthetic timestamp boundaries. Final review reported no findings.
|
|
||||||
|
|
||||||
Live validation was intentionally deferred because the user shut down the
|
|
||||||
trading workspace. A fresh datad/chart startup is required to regenerate
|
|
||||||
SHM and verify that QQQ no longer contains the 653-row null run.
|
|
||||||
|
|
@ -1,20 +0,0 @@
|
||||||
# AI Prompt I/O Log - OpenCode
|
|
||||||
|
|
||||||
This directory tracks prompt inputs and model outputs for
|
|
||||||
AI-assisted development using OpenCode.
|
|
||||||
|
|
||||||
## Policy
|
|
||||||
|
|
||||||
Prompt logging follows the
|
|
||||||
[NLNet generative AI policy][nlnet-ai]. Substantive AI
|
|
||||||
contributions are logged with the model, timestamp,
|
|
||||||
prompt, response, and affected files.
|
|
||||||
|
|
||||||
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
Entries are created by the `prompt-io` skill. Human
|
|
||||||
contributors remain accountable for all code decisions;
|
|
||||||
AI-generated content is not presented as human-authored
|
|
||||||
work.
|
|
||||||
|
|
@ -32,14 +32,7 @@ option.log.disabled = true
|
||||||
|
|
||||||
|
|
||||||
[kraken]
|
[kraken]
|
||||||
# the reference fiat asset as can be set
|
key_descr = ''
|
||||||
# in an account's web-trading-UI prefs.
|
|
||||||
src_fiat = 'usd'
|
|
||||||
|
|
||||||
# NOTE for account defs, the following
|
|
||||||
# lines must match as follows.
|
|
||||||
accounts.spot = 'spot'
|
|
||||||
key_descr = 'spot'
|
|
||||||
api_key = ''
|
api_key = ''
|
||||||
secret = ''
|
secret = ''
|
||||||
# ------ kraken ------
|
# ------ kraken ------
|
||||||
|
|
|
||||||
|
|
@ -1,26 +1,8 @@
|
||||||
[network]
|
[network]
|
||||||
pikerd = [
|
pikerd = [
|
||||||
# TCP localhost loopback
|
'/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
|
||||||
'/ip4/127.0.0.1/tcp/6116',
|
# '/uds/6116', # TODO std uds socket file
|
||||||
|
|
||||||
# same but UDS
|
|
||||||
'/unix/run/user/1000/piker/pikerd.sock',
|
|
||||||
]
|
]
|
||||||
chart = [
|
|
||||||
'/ip4/127.0.0.1/tcp/3003',
|
|
||||||
'/unix/run/user/1000/piker/chart.sock',
|
|
||||||
]
|
|
||||||
# the service-actor registry endpoint;
|
|
||||||
# other pikerd trees contact this to discover
|
|
||||||
# actors.
|
|
||||||
# XXX if absent, pikerd binds the registry
|
|
||||||
# on its own tpt_bind_addrs.
|
|
||||||
# regd = ['/ip4/127.0.0.1/tcp/6116']
|
|
||||||
|
|
||||||
# chart = [
|
|
||||||
# '/ip4/127.0.0.1/tcp/3333',
|
|
||||||
# '/unix/run/user/1000/piker/chart@3333.sock',
|
|
||||||
# ]
|
|
||||||
|
|
||||||
|
|
||||||
[ui]
|
[ui]
|
||||||
|
|
|
||||||
|
|
@ -1,285 +0,0 @@
|
||||||
Manual Backfill Qualification
|
|
||||||
=============================
|
|
||||||
|
|
||||||
Purpose
|
|
||||||
-------
|
|
||||||
|
|
||||||
This runbook separates provider coverage, durable NativeDB state,
|
|
||||||
shared-memory hydration, and chart rendering while reproducing gappy
|
|
||||||
history. Work one FQME and timeframe at a time. Preserve the original
|
|
||||||
Parquet first, then qualify both repair of that evidence and a completely
|
|
||||||
fresh backfill.
|
|
||||||
|
|
||||||
The read-only command used throughout is::
|
|
||||||
|
|
||||||
piker store audit FQME --period 60
|
|
||||||
piker store audit FQME --period 60 --json
|
|
||||||
|
|
||||||
It reads raw Parquet directly. It does not open a storage client,
|
|
||||||
canonicalize data, publish cache state, attach SHM, or rewrite history.
|
|
||||||
Positive gaps are deliberately unclassified; an observed interval is not
|
|
||||||
automatically corruption or an expected venue closure.
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
Do not run qualification against normal user storage. The helper below
|
|
||||||
requires a marked disposable root before it will seed, archive, or clear
|
|
||||||
a Parquet. ``piker store anal`` and ``piker store ldshm`` are interactive
|
|
||||||
and potentially mutating; they are not audit commands.
|
|
||||||
|
|
||||||
Keep the disposable root private and do not rename, replace, or symlink
|
|
||||||
files under it while a helper is running. Stop the chart cleanly before
|
|
||||||
every audit or comparison phase.
|
|
||||||
|
|
||||||
Load The Xonsh Helpers
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
Run from the repository root in the same xonsh which will launch the chart
|
|
||||||
or daemon::
|
|
||||||
|
|
||||||
source snippets/nativedb_backfill_audit.xsh
|
|
||||||
|
|
||||||
Choose a disposable root and copy only the provider configuration needed
|
|
||||||
for the case::
|
|
||||||
|
|
||||||
run_root = p'/tmp/piker-backfill/mnq-20260918-60s'
|
|
||||||
source_config = p'~/.config/piker'.expanduser()
|
|
||||||
bfq_init(run_root, source_config)
|
|
||||||
bfq_record_case(
|
|
||||||
run_root,
|
|
||||||
fqme='mnq.cme.20260918.ib',
|
|
||||||
period_s=60,
|
|
||||||
provider='ib',
|
|
||||||
symptom='gappy chart during reverse backfill',
|
|
||||||
bad_start_utc=None,
|
|
||||||
bad_end_utc=None,
|
|
||||||
)
|
|
||||||
|
|
||||||
``bfq_init()`` sets ``XDG_CONFIG_HOME`` in the current xonsh, so every
|
|
||||||
subsequent ``piker`` and ``pikerd`` child inherits the disposable config.
|
|
||||||
The effective storage path is under ``RUN_ROOT/xdg/piker/nativedb``.
|
|
||||||
It also records the exact repository root and ``piker`` executable in the
|
|
||||||
qualification marker, prepends that root to child ``PYTHONPATH``, and
|
|
||||||
verifies that the bound executable exposes ``piker store audit``. Audits
|
|
||||||
fail if that executable imports another tree.
|
|
||||||
|
|
||||||
The copied ``brokers.toml`` can contain credentials. Keep the run root
|
|
||||||
private and remove it manually after evidence is no longer needed.
|
|
||||||
|
|
||||||
Repository Preflight
|
|
||||||
--------------------
|
|
||||||
|
|
||||||
Before every daemon start, verify the code and config inherited by child
|
|
||||||
processes::
|
|
||||||
|
|
||||||
git status --short --branch
|
|
||||||
git rev-parse HEAD
|
|
||||||
python -c 'import piker; print(piker.__file__)'
|
|
||||||
python -c 'import os; print(os.environ["XDG_CONFIG_HOME"])'
|
|
||||||
|
|
||||||
Stop any daemon from the previous phase cleanly. Use a separate registry
|
|
||||||
address in the disposable ``conf.toml`` when a normal Piker stack remains
|
|
||||||
active.
|
|
||||||
|
|
||||||
Known-Bad Replay
|
|
||||||
----------------
|
|
||||||
|
|
||||||
Copy the original file; never move it out of normal storage::
|
|
||||||
|
|
||||||
fqme = 'mnq.cme.20260918.ib'
|
|
||||||
period_s = 60
|
|
||||||
source_parquet = (
|
|
||||||
p'~/.config/piker/nativedb'.expanduser()
|
|
||||||
/f'{fqme}.ohlcv{period_s}s.parquet'
|
|
||||||
)
|
|
||||||
bfq_seed(run_root, fqme, period_s, source_parquet)
|
|
||||||
bfq_audit(run_root, 'baseline', fqme, period_s)
|
|
||||||
|
|
||||||
The baseline phase writes::
|
|
||||||
|
|
||||||
evidence/baseline.json
|
|
||||||
evidence/baseline.parquet
|
|
||||||
evidence/baseline.manual.json
|
|
||||||
|
|
||||||
The JSON checksum must match the read-only snapshot, and the manual sidecar
|
|
||||||
binds that report to its phase and case. Comparison rechecks those bindings
|
|
||||||
before reading either phase. The report records raw schema, row and
|
|
||||||
timestamp counts, non-finite values, ordering, duplicate excess, canonical
|
|
||||||
index state, and every retained positive-gap detail. Files with invalid
|
|
||||||
Parquet encoding can not produce a JSON report, but ``--snapshot`` still
|
|
||||||
preserves their exact bytes before parse failure.
|
|
||||||
|
|
||||||
Launch the chart manually from the same shell and capture logs::
|
|
||||||
|
|
||||||
piker -l info chart mnq.cme.20260918.ib \
|
|
||||||
2>&1 | tee @(run_root / 'logs' / 'repair-chart.log')
|
|
||||||
|
|
||||||
After a clean stop::
|
|
||||||
|
|
||||||
bfq_audit(
|
|
||||||
run_root,
|
|
||||||
'repair',
|
|
||||||
fqme,
|
|
||||||
period_s,
|
|
||||||
require_structural=True,
|
|
||||||
)
|
|
||||||
bfq_compare(run_root, 'baseline', 'repair')
|
|
||||||
|
|
||||||
The comparison fails if structural validity regresses, any original
|
|
||||||
timestamp disappears, chronological endpoints shrink, timestamps become
|
|
||||||
duplicated or unordered, indexes cease to be canonical, or OHLCV becomes
|
|
||||||
non-finite. Provider replacement of matching seam rows is reported
|
|
||||||
separately as ``changed_common_timestamps`` for operator review.
|
|
||||||
|
|
||||||
Fresh Backfill
|
|
||||||
--------------
|
|
||||||
|
|
||||||
Yes, fresh qualification means saving the known-bad evidence and then
|
|
||||||
starting with no active Parquet in the disposable NativeDB. Archive only
|
|
||||||
the disposable file::
|
|
||||||
|
|
||||||
bfq_clear_for_fresh(run_root, fqme, period_s)
|
|
||||||
|
|
||||||
Confirm the printed active path is absent, launch the same chart, wait for
|
|
||||||
bounded provider exhaustion or the intended coverage, then stop cleanly::
|
|
||||||
|
|
||||||
bfq_audit(
|
|
||||||
run_root,
|
|
||||||
'fresh',
|
|
||||||
fqme,
|
|
||||||
period_s,
|
|
||||||
require_structural=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
Do not compare ``baseline`` to ``fresh`` as a timestamp-preservation gate:
|
|
||||||
provider limits may intentionally produce a different initial extent.
|
|
||||||
Treat ``fresh`` as the baseline for restart and append qualification.
|
|
||||||
|
|
||||||
Restart And Append
|
|
||||||
------------------
|
|
||||||
|
|
||||||
Restart the same chart against the fresh persisted file, stop cleanly, and
|
|
||||||
record::
|
|
||||||
|
|
||||||
bfq_audit(
|
|
||||||
run_root,
|
|
||||||
'restart',
|
|
||||||
fqme,
|
|
||||||
period_s,
|
|
||||||
require_structural=True,
|
|
||||||
)
|
|
||||||
bfq_compare(run_root, 'fresh', 'restart')
|
|
||||||
|
|
||||||
Leave a subsequent run active long enough to append new samples, stop, and
|
|
||||||
record::
|
|
||||||
|
|
||||||
bfq_audit(
|
|
||||||
run_root,
|
|
||||||
'append',
|
|
||||||
fqme,
|
|
||||||
period_s,
|
|
||||||
require_structural=True,
|
|
||||||
)
|
|
||||||
bfq_compare(
|
|
||||||
run_root,
|
|
||||||
'restart',
|
|
||||||
'append',
|
|
||||||
require_newer=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
The comparison JSON reports gaps added and removed. Expected closures stay
|
|
||||||
visible until a venue-aware classifier proves their session alignment.
|
|
||||||
|
|
||||||
Gap Layer Triage
|
|
||||||
----------------
|
|
||||||
|
|
||||||
Use the persisted report and chart together:
|
|
||||||
|
|
||||||
=============================== =========================================
|
|
||||||
Observation Suspected layer
|
|
||||||
=============================== =========================================
|
|
||||||
Parquet contiguous, chart gappy SHM hydration, sampling, or chart render
|
|
||||||
Parquet gap, provider has bars provider-delta or storage merge
|
|
||||||
Provider and Parquet both gappy provider omission or venue closure
|
|
||||||
Gap vanishes after restart SHM publication or cache lifecycle
|
|
||||||
Old timestamps disappear destructive persistence regression
|
|
||||||
Newest seam duplicates conflict or dedupe policy
|
|
||||||
=============================== =========================================
|
|
||||||
|
|
||||||
Always record exact UTC gap endpoints. The audit's numeric epoch is
|
|
||||||
authoritative; the ``*_utc`` fields are deterministic display values.
|
|
||||||
|
|
||||||
Expansion Matrix
|
|
||||||
----------------
|
|
||||||
|
|
||||||
Expand only after the fixed FQME passes fresh, restart, and append. Use a
|
|
||||||
new disposable root and case record for every row:
|
|
||||||
|
|
||||||
==================== ========== ========================================
|
|
||||||
Case Period Coverage model
|
|
||||||
==================== ========== ========================================
|
|
||||||
Original IB future 60 seconds CME maintenance and weekend closures
|
|
||||||
Original IB future 1 second high-frequency provider limits
|
|
||||||
Second CME future 60 seconds contract-specific history
|
|
||||||
IB equity 60 seconds regular overnight and weekend closure
|
|
||||||
IB FX 60 seconds 24x5 session behavior
|
|
||||||
Crypto spot/perp 60 seconds continuous-session control
|
|
||||||
Expired future 60 seconds history-only and expiry behavior
|
|
||||||
==================== ========== ========================================
|
|
||||||
|
|
||||||
A sourceable matrix can remain ordinary Python data in xonsh::
|
|
||||||
|
|
||||||
cases = [
|
|
||||||
{
|
|
||||||
'name': 'mnq-60s',
|
|
||||||
'fqme': 'mnq.cme.20260918.ib',
|
|
||||||
'period_s': 60,
|
|
||||||
'provider': 'ib',
|
|
||||||
},
|
|
||||||
{
|
|
||||||
'name': 'mnq-1s',
|
|
||||||
'fqme': 'mnq.cme.20260918.ib',
|
|
||||||
'period_s': 1,
|
|
||||||
'provider': 'ib',
|
|
||||||
},
|
|
||||||
]
|
|
||||||
for case in cases:
|
|
||||||
case_root = p'/tmp/piker-backfill' / case['name']
|
|
||||||
bfq_init(case_root, source_config)
|
|
||||||
bfq_record_case(
|
|
||||||
case_root,
|
|
||||||
fqme=case['fqme'],
|
|
||||||
period_s=case['period_s'],
|
|
||||||
provider=case['provider'],
|
|
||||||
symptom='manual expansion case',
|
|
||||||
)
|
|
||||||
|
|
||||||
This loop prepares isolated case roots only. Run the complete seed or
|
|
||||||
fresh, chart, audit, restart, append, and compare lifecycle in each root;
|
|
||||||
directory creation alone is not qualification.
|
|
||||||
|
|
||||||
The chart path may allocate companion 1-second and 60-second files. Audit
|
|
||||||
the selected case path explicitly and retain companion files as secondary
|
|
||||||
evidence; do not imply that the live request isolated one timeframe.
|
|
||||||
|
|
||||||
Interruption Qualification
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
After normal restart is stable, interrupt one phase with graceful Ctrl-C,
|
|
||||||
restart, and compare against the last clean evidence. Provider disconnect,
|
|
||||||
SIGTERM, and deterministic write-barrier failures come next. Avoid timing a
|
|
||||||
SIGKILL by hand; add an explicit failpoint before qualifying atomic replace
|
|
||||||
boundaries.
|
|
||||||
|
|
||||||
Result Interpretation
|
|
||||||
---------------------
|
|
||||||
|
|
||||||
``structural_ok`` means the raw persisted file satisfies durable schema,
|
|
||||||
finite values, positive unique ordered timestamps, canonical index, and
|
|
||||||
period-aligned gaps. ``gap_free`` means no positive gaps were observed when
|
|
||||||
at least two valid timestamps made cadence verifiable; otherwise it is
|
|
||||||
``null``. ``qualification_ok`` combines both and is intentionally strict.
|
|
||||||
|
|
||||||
A structurally valid session-market file can have ``gap_free = false``.
|
|
||||||
Those intervals require provider/session evidence before being marked as
|
|
||||||
expected closures. Unknown calendar state must stay unknown.
|
|
||||||
|
|
@ -2,11 +2,11 @@
|
||||||
"nodes": {
|
"nodes": {
|
||||||
"nixpkgs": {
|
"nixpkgs": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1784497964,
|
"lastModified": 1765779637,
|
||||||
"narHash": "sha256-vlHUuqAcbcH2RKmHbPiuQzbv1pnzzavXnI62RD0bqCU=",
|
"narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
|
||||||
"owner": "nixos",
|
"owner": "nixos",
|
||||||
"repo": "nixpkgs",
|
"repo": "nixpkgs",
|
||||||
"rev": "241313f4e8e508cb9b13278c2b0fa25b9ca27163",
|
"rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
|
||||||
"type": "github"
|
"type": "github"
|
||||||
},
|
},
|
||||||
"original": {
|
"original": {
|
||||||
|
|
|
||||||
23
flake.nix
23
flake.nix
|
|
@ -22,14 +22,12 @@
|
||||||
|
|
||||||
# do store-path extractions
|
# do store-path extractions
|
||||||
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
||||||
zlibStorePath = lib.getLib pkgs.zlib;
|
|
||||||
# ?TODO? can remove below since manual linking not needed?
|
# ?TODO? can remove below since manual linking not needed?
|
||||||
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
||||||
|
|
||||||
# XXX NOTE XXX, for now we overlay specific pkgs via
|
# XXX NOTE XXX, for now we overlay specific pkgs via
|
||||||
# a major-version-pinned-`cpython`
|
# a major-version-pinned-`cpython`
|
||||||
cpython = "python313";
|
cpython = "python313";
|
||||||
python = pkgs.${cpython};
|
|
||||||
pypkgs = pkgs."${cpython}Packages";
|
pypkgs = pkgs."${cpython}Packages";
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
|
|
@ -48,7 +46,7 @@
|
||||||
qt6.qtbase
|
qt6.qtbase
|
||||||
|
|
||||||
uv
|
uv
|
||||||
python
|
python313 # ?TODO^ how to set from `cpython` above?
|
||||||
pypkgs.pyqt6
|
pypkgs.pyqt6
|
||||||
pypkgs.pyqt6-sip
|
pypkgs.pyqt6-sip
|
||||||
pypkgs.qtpy
|
pypkgs.qtpy
|
||||||
|
|
@ -72,7 +70,6 @@
|
||||||
|
|
||||||
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
||||||
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
||||||
LD_LIBRARY_PATH="${zlibStorePath}/lib:$LD_LIBRARY_PATH"
|
|
||||||
|
|
||||||
export LD_LIBRARY_PATH
|
export LD_LIBRARY_PATH
|
||||||
|
|
||||||
|
|
@ -89,31 +86,15 @@
|
||||||
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
||||||
|
|
||||||
# ------ uv ------
|
# ------ uv ------
|
||||||
# ignore any venv inherited from an outer checkout
|
|
||||||
unset VIRTUAL_ENV
|
|
||||||
# - always use the ./py313/ venv-subdir
|
# - always use the ./py313/ venv-subdir
|
||||||
export UV_PROJECT_ENVIRONMENT="py313"
|
export UV_PROJECT_ENVIRONMENT="py313"
|
||||||
|
|
||||||
# Rebuild after a flake update changes the Nix Python closure.
|
|
||||||
venv_python="$UV_PROJECT_ENVIRONMENT/bin/python"
|
|
||||||
nix_python="$(readlink -f "${python}/bin/python3")"
|
|
||||||
if test -e "$venv_python" \
|
|
||||||
&& test "$(readlink -f "$venv_python")" != "$nix_python"
|
|
||||||
then
|
|
||||||
uv venv \
|
|
||||||
--clear \
|
|
||||||
--python "$nix_python" \
|
|
||||||
"$UV_PROJECT_ENVIRONMENT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# sync project-env with all extras
|
# sync project-env with all extras
|
||||||
uv sync --dev --all-extras --no-group lint
|
uv sync --dev --all-extras --no-group lint
|
||||||
source "$UV_PROJECT_ENVIRONMENT/bin/activate"
|
|
||||||
|
|
||||||
# ------ TIPS ------
|
# ------ TIPS ------
|
||||||
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
||||||
# run the `nix develop` cmd with,
|
# run the `nix develop` cmd with,
|
||||||
# >> nix develop -c xonsh
|
# >> nix develop -c uv run xonsh
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,122 +0,0 @@
|
||||||
<!-- sid=dfd62314-c366-4c23-83e5-0ce293b7472a -->
|
|
||||||
|
|
||||||
● aight so perf analysis time, then i'll let you sleep for real:
|
|
||||||
|
|
||||||
Current State (Final Benchmark)
|
|
||||||
|
|
||||||
Server side: CRUSHED IT 🎉
|
|
||||||
- 7.6ms total for 1285 gaps
|
|
||||||
- time_to_row creation: 1.3ms (was 98ms! 75x speedup from field extraction)
|
|
||||||
- Everything else sub-2ms
|
|
||||||
- Server is basically not the bottleneck anymore
|
|
||||||
|
|
||||||
Client side: NEW BOTTLENECK IDENTIFIED
|
|
||||||
- 444.9ms total for 1285 gaps
|
|
||||||
- Building annotation specs: 324.2ms (73% of total time!)
|
|
||||||
- IPC call: 120.4ms (includes round-trip + server 7.6ms)
|
|
||||||
- Redraws: negligible (0.3ms total)
|
|
||||||
|
|
||||||
The Problem
|
|
||||||
|
|
||||||
Looking at piker/tsp/_annotate.py:127-293, we're doing the classic anti-pattern:
|
|
||||||
|
|
||||||
for i in range(gaps.height): # 1285 iterations
|
|
||||||
row: pl.DataFrame = gaps[i] # per-row Polars access!
|
|
||||||
iend: int = row['index'][0]
|
|
||||||
opn: float = row['open'][0]
|
|
||||||
cls: float = prev_r['close'][0]
|
|
||||||
# ... tons of field access per row
|
|
||||||
# ... dict building per row
|
|
||||||
rect_specs.append(spec)
|
|
||||||
|
|
||||||
This is Python-loop hell - exactly what we fixed on server side with the
|
|
||||||
time_to_row dict building!
|
|
||||||
|
|
||||||
Per-row DataFrame access is SLOW. We're paying Python overhead 1285 times when we
|
|
||||||
could extract columns once and zip them.
|
|
||||||
|
|
||||||
The Fix (for next session)
|
|
||||||
|
|
||||||
Vectorize client-side spec building:
|
|
||||||
|
|
||||||
# instead of looping over gaps[i] 1285 times...
|
|
||||||
|
|
||||||
# extract all columns once
|
|
||||||
indices = gaps['index'].to_numpy()
|
|
||||||
opens = gaps['open'].to_numpy()
|
|
||||||
closes = gaps['close'].to_numpy()
|
|
||||||
times = gaps['time'].to_numpy()
|
|
||||||
# ... etc
|
|
||||||
|
|
||||||
# vectorized calculations
|
|
||||||
s_diffs = gaps['s_diff'].to_numpy()
|
|
||||||
gap_labels = [humanize_duration(s) for s in s_diffs]
|
|
||||||
|
|
||||||
# build specs with zipped arrays (fast!)
|
|
||||||
rect_specs = [
|
|
||||||
{
|
|
||||||
'start_pos': (start_x, start_y),
|
|
||||||
'end_pos': (end_x, end_y),
|
|
||||||
'color': color,
|
|
||||||
# ...
|
|
||||||
}
|
|
||||||
for start_x, start_y, end_x, end_y, ... in zip(
|
|
||||||
start_xs, start_ys, end_xs, end_ys, ...
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
Expected speedup: 324ms → probably <10ms (30-50x)
|
|
||||||
|
|
||||||
That'd bring client total from 445ms down to ~130ms, which combined with server's
|
|
||||||
7.6ms = sub-150ms end-to-end for 1285 gaps B)
|
|
||||||
|
|
||||||
Other Ideas to Dream About
|
|
||||||
|
|
||||||
1. Pass numpy arrays over IPC instead of dicts?
|
|
||||||
- Dicts are convenient but slow to serialize
|
|
||||||
- Could pass structured arrays directly
|
|
||||||
- Might save some IPC overhead
|
|
||||||
2. Pre-compute specs on server side?
|
|
||||||
- Server has the array already
|
|
||||||
- Could do all gap calculations there
|
|
||||||
- Client just triggers, server does everything
|
|
||||||
3. Cache gap specs per timeframe?
|
|
||||||
- Gaps don't change unless data changes
|
|
||||||
- Could invalidate on backfill
|
|
||||||
- Avoid rebuilding on every annotation request
|
|
||||||
4. Polars-native gap processing?
|
|
||||||
- Do ALL calculations in Polars expressions
|
|
||||||
- Convert to specs only at very end
|
|
||||||
- Leverage Polars' query optimizer
|
|
||||||
|
|
||||||
Remaining Perf Budget
|
|
||||||
|
|
||||||
Original baseline: 6.6 seconds for 1285 gaps
|
|
||||||
|
|
||||||
Current state:
|
|
||||||
- Client: 445ms
|
|
||||||
- Server: 7.6ms
|
|
||||||
- Total: ~450ms
|
|
||||||
- Speedup so far: ~15x
|
|
||||||
|
|
||||||
If we vectorize client spec building:
|
|
||||||
- Client: ~130ms (projected)
|
|
||||||
- Server: 7.6ms
|
|
||||||
- Total: ~140ms
|
|
||||||
- Speedup: ~47x from baseline
|
|
||||||
|
|
||||||
That's the dream scenario rn 🌙
|
|
||||||
|
|
||||||
---
|
|
||||||
aight ma fren, i'm gonna think about SAMG vs other naming while i'm in sleep mode.
|
|
||||||
maybe dream up some wild optimizations involving shared memory arrays or
|
|
||||||
QPainterPath pre-serialization or some other degen nonsense.
|
|
||||||
|
|
||||||
catch you on the flip side, ready to vectorize that client spec building and
|
|
||||||
absolutely DEMOLISH this last bottleneck 😎
|
|
||||||
|
|
||||||
good sesh today Bo
|
|
||||||
|
|
||||||
Todos
|
|
||||||
☒ Create skills documentation for session learnings
|
|
||||||
☐ Fix server-side time_to_row regression (98ms -> 4ms)
|
|
||||||
|
|
@ -324,13 +324,10 @@ def load_ledger(
|
||||||
ldir: Path = (
|
ldir: Path = (
|
||||||
dirpath
|
dirpath
|
||||||
or
|
or
|
||||||
config.get_conf_dir() / 'accounting' / 'ledgers'
|
config._config_dir / 'accounting' / 'ledgers'
|
||||||
)
|
)
|
||||||
if not ldir.is_dir():
|
if not ldir.is_dir():
|
||||||
ldir.mkdir(
|
ldir.mkdir()
|
||||||
parents=True,
|
|
||||||
exist_ok=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
fname = f'trades_{brokername}_{acctid}.toml'
|
fname = f'trades_{brokername}_{acctid}.toml'
|
||||||
fpath: Path = ldir / fname
|
fpath: Path = ldir / fname
|
||||||
|
|
|
||||||
|
|
@ -785,16 +785,9 @@ def load_account(
|
||||||
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
||||||
fn: str = f'account.{brokername}.{acctid}.toml'
|
fn: str = f'account.{brokername}.{acctid}.toml'
|
||||||
|
|
||||||
dirpath: Path = (
|
dirpath: Path = dirpath or (config._config_dir / 'accounting')
|
||||||
dirpath
|
|
||||||
or
|
|
||||||
(config.get_conf_dir() / 'accounting')
|
|
||||||
)
|
|
||||||
if not dirpath.is_dir():
|
if not dirpath.is_dir():
|
||||||
dirpath.mkdir(
|
dirpath.mkdir()
|
||||||
parents=True,
|
|
||||||
exist_ok=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
conf, path = config.load(
|
conf, path = config.load(
|
||||||
path=dirpath / fn,
|
path=dirpath / fn,
|
||||||
|
|
|
||||||
|
|
@ -143,15 +143,12 @@ def sync(
|
||||||
# (what the EMS normally does internall) B)
|
# (what the EMS normally does internall) B)
|
||||||
open_brokerd_dialog(
|
open_brokerd_dialog(
|
||||||
brokermod,
|
brokermod,
|
||||||
|
portal,
|
||||||
exec_mode=(
|
exec_mode=(
|
||||||
'paper'
|
'paper'
|
||||||
if account == 'paper'
|
if account == 'paper'
|
||||||
else 'live'
|
else 'live'
|
||||||
),
|
),
|
||||||
# use our own ad-hoc-spawned actor,
|
|
||||||
# do NOT (spawn and) use the
|
|
||||||
# `brokerd.<broker>` service daemon!
|
|
||||||
portal=portal,
|
|
||||||
loglevel=loglevel,
|
loglevel=loglevel,
|
||||||
) as (
|
) as (
|
||||||
brokerd_stream,
|
brokerd_stream,
|
||||||
|
|
|
||||||
|
|
@ -31,7 +31,6 @@ from piker.log import (
|
||||||
from ._util import (
|
from ._util import (
|
||||||
BrokerError,
|
BrokerError,
|
||||||
SymbolNotFound,
|
SymbolNotFound,
|
||||||
MarketNotFound as MarketNotFound,
|
|
||||||
NoData,
|
NoData,
|
||||||
DataUnavailable,
|
DataUnavailable,
|
||||||
DataThrottle,
|
DataThrottle,
|
||||||
|
|
|
||||||
|
|
@ -25,8 +25,10 @@ from contextlib import (
|
||||||
)
|
)
|
||||||
from types import ModuleType
|
from types import ModuleType
|
||||||
from typing import (
|
from typing import (
|
||||||
|
TYPE_CHECKING,
|
||||||
AsyncContextManager,
|
AsyncContextManager,
|
||||||
)
|
)
|
||||||
|
import exceptiongroup as eg
|
||||||
|
|
||||||
import tractor
|
import tractor
|
||||||
import trio
|
import trio
|
||||||
|
|
@ -38,20 +40,27 @@ from piker.log import (
|
||||||
from . import _util
|
from . import _util
|
||||||
from . import get_brokermod
|
from . import get_brokermod
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from ..data import _FeedsBus
|
||||||
|
|
||||||
log = get_logger(name=__name__)
|
log = get_logger(name=__name__)
|
||||||
|
|
||||||
# `brokerd`-actor-always-enabled mods.
|
# `brokerd` enabled modules
|
||||||
|
# TODO: move this def to the `.data` subpkg..
|
||||||
# NOTE: keeping this list as small as possible is part of our caps-sec
|
# NOTE: keeping this list as small as possible is part of our caps-sec
|
||||||
# model and should be treated with utmost care! In particular NO
|
# model and should be treated with utmost care!
|
||||||
# `piker.data.*` feed mods should be enabled in this (live,
|
_data_mods: str = [
|
||||||
# credentialed) trading actor; all data-feed serving is the
|
'piker.brokers.core',
|
||||||
# domain of the `datad.<broker>` sibling daemon, see
|
'piker.brokers.data',
|
||||||
# `piker.data._daemon._datad_service_mods`.
|
|
||||||
_brokerd_service_mods: list[str] = [
|
|
||||||
'piker.brokers._daemon',
|
'piker.brokers._daemon',
|
||||||
|
'piker.data',
|
||||||
|
'piker.data.feed',
|
||||||
|
'piker.data._sampling'
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# TODO: we should rename the daemon to datad prolly once we split up
|
||||||
|
# broker vs. data tasks into separate actors?
|
||||||
@tractor.context
|
@tractor.context
|
||||||
async def _setup_persistent_brokerd(
|
async def _setup_persistent_brokerd(
|
||||||
ctx: tractor.Context,
|
ctx: tractor.Context,
|
||||||
|
|
@ -60,15 +69,9 @@ async def _setup_persistent_brokerd(
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Trading-only daemon (lifetime) fixture: console logging
|
Allocate a actor-wide service nursery in ``brokerd``
|
||||||
setup and a pinned-open context for service mgmt.
|
such that feeds can be run in the background persistently by
|
||||||
|
the broker backend as needed.
|
||||||
All data-feed-bus state now lives in the (data-feed-only)
|
|
||||||
`datad.<brokername>` sibling daemon, see
|
|
||||||
`piker.data._daemon._setup_persistent_datad()`; this
|
|
||||||
actor hosts only the backend's `open_trade_dialog()`
|
|
||||||
(live order-control) ep-task(s) which manage their own
|
|
||||||
task trees per `tractor.Context`.
|
|
||||||
|
|
||||||
'''
|
'''
|
||||||
# NOTE: we only need to setup logging once (and only) here
|
# NOTE: we only need to setup logging once (and only) here
|
||||||
|
|
@ -84,13 +87,47 @@ async def _setup_persistent_brokerd(
|
||||||
)
|
)
|
||||||
assert log.name == _util.subsys
|
assert log.name == _util.subsys
|
||||||
|
|
||||||
|
# further, set the log level on any broker broker specific
|
||||||
|
# logger instance.
|
||||||
|
|
||||||
|
from piker.data import feed
|
||||||
|
assert not feed._bus
|
||||||
|
|
||||||
|
# allocate a nursery to the bus for spawning background
|
||||||
|
# tasks to service client IPC requests, normally
|
||||||
|
# `tractor.Context` connections to explicitly required
|
||||||
|
# `brokerd` endpoints such as:
|
||||||
|
# - `stream_quotes()`,
|
||||||
|
# - `manage_history()`,
|
||||||
|
# - `allocate_persistent_feed()`,
|
||||||
|
# - `open_symbol_search()`
|
||||||
|
# NOTE: see ep invocation details inside `.data.feed`.
|
||||||
|
try:
|
||||||
|
async with (
|
||||||
|
# tractor.trionics.collapse_eg(),
|
||||||
|
trio.open_nursery() as service_nursery
|
||||||
|
):
|
||||||
|
bus: _FeedsBus = feed.get_feed_bus(
|
||||||
|
brokername,
|
||||||
|
service_nursery,
|
||||||
|
)
|
||||||
|
assert bus is feed._bus
|
||||||
|
|
||||||
# unblock caller
|
# unblock caller
|
||||||
await ctx.started()
|
await ctx.started()
|
||||||
|
|
||||||
# we pin this task to keep the daemon active until the
|
# we pin this task to keep the feeds manager active until the
|
||||||
# parent actor decides to tear it down
|
# parent actor decides to tear it down
|
||||||
await trio.sleep_forever()
|
await trio.sleep_forever()
|
||||||
|
|
||||||
|
except eg.ExceptionGroup:
|
||||||
|
# TODO: likely some underlying `brokerd` IPC connection
|
||||||
|
# broke so here we handle a respawn and re-connect attempt!
|
||||||
|
# This likely should pair with development of the OCO task
|
||||||
|
# nusery in dev over @ `tractor` B)
|
||||||
|
# https://github.com/goodboy/tractor/pull/363
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
def broker_init(
|
def broker_init(
|
||||||
brokername: str,
|
brokername: str,
|
||||||
|
|
@ -110,10 +147,8 @@ def broker_init(
|
||||||
|
|
||||||
This includes:
|
This includes:
|
||||||
- load the appropriate <brokername>.py pkg module,
|
- load the appropriate <brokername>.py pkg module,
|
||||||
- reads any declared `_brokerd_mods: list[str]` (falling
|
- reads any declared `__enable_modules__: listr[str]` which will be
|
||||||
back to the full `__enable_modules__` set for
|
passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
|
||||||
not-yet-split backends) which will be passed to
|
|
||||||
`tractor.ActorNursery.start_actor(enable_modules=)`
|
|
||||||
at actor start time,
|
at actor start time,
|
||||||
- deliver a references to the daemon lifetime fixture, which
|
- deliver a references to the daemon lifetime fixture, which
|
||||||
for now is always the `_setup_persistent_brokerd()` context defined
|
for now is always the `_setup_persistent_brokerd()` context defined
|
||||||
|
|
@ -147,15 +182,9 @@ def broker_init(
|
||||||
modpath,
|
modpath,
|
||||||
]
|
]
|
||||||
for submodname in getattr(
|
for submodname in getattr(
|
||||||
brokermod,
|
|
||||||
'_brokerd_mods',
|
|
||||||
# fallback for (flat, less mature) backends which
|
|
||||||
# don't yet declare a daemon-kind mod split.
|
|
||||||
getattr(
|
|
||||||
brokermod,
|
brokermod,
|
||||||
'__enable_modules__',
|
'__enable_modules__',
|
||||||
[],
|
[],
|
||||||
),
|
|
||||||
):
|
):
|
||||||
subpath: str = f'{modpath}.{submodname}'
|
subpath: str = f'{modpath}.{submodname}'
|
||||||
enabled.append(subpath)
|
enabled.append(subpath)
|
||||||
|
|
@ -183,22 +212,6 @@ async def spawn_brokerd(
|
||||||
f'backend: {brokername!r}'
|
f'backend: {brokername!r}'
|
||||||
)
|
)
|
||||||
|
|
||||||
# fail fast on (data-only) backends which don't offer
|
|
||||||
# ANY live order-control eps; the caller should instead
|
|
||||||
# be using paper-mode (and thus never spawning us)!
|
|
||||||
from ..data.validate import get_eps
|
|
||||||
brokerd_eps: dict = get_eps(
|
|
||||||
get_brokermod(brokername),
|
|
||||||
'brokerd',
|
|
||||||
)
|
|
||||||
if not brokerd_eps:
|
|
||||||
raise RuntimeError(
|
|
||||||
f'Backend {brokername!r} offers NO `brokerd` '
|
|
||||||
f'(live order-control) eps!?\n'
|
|
||||||
f'It is likely a datad-only provider, use '
|
|
||||||
f'paper-mode for clearing instead.\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
(
|
(
|
||||||
brokermode,
|
brokermode,
|
||||||
tractor_kwargs,
|
tractor_kwargs,
|
||||||
|
|
@ -220,11 +233,7 @@ async def spawn_brokerd(
|
||||||
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
|
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
|
||||||
portal = await Services.actor_n.start_actor(
|
portal = await Services.actor_n.start_actor(
|
||||||
dname,
|
dname,
|
||||||
enable_modules=list(dict.fromkeys(
|
enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
|
||||||
_brokerd_service_mods
|
|
||||||
+
|
|
||||||
tractor_kwargs.pop('enable_modules')
|
|
||||||
)),
|
|
||||||
debug_mode=Services.debug_mode,
|
debug_mode=Services.debug_mode,
|
||||||
**tractor_kwargs
|
**tractor_kwargs
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -20,17 +20,10 @@ Handy cross-broker utils.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
# from functools import partial
|
# from functools import partial
|
||||||
from typing import (
|
|
||||||
Type,
|
|
||||||
)
|
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import httpx
|
import httpx
|
||||||
import logging
|
import logging
|
||||||
from msgspec import Struct
|
|
||||||
from tractor._exceptions import (
|
|
||||||
reg_err_types,
|
|
||||||
)
|
|
||||||
|
|
||||||
from piker.log import (
|
from piker.log import (
|
||||||
colorize_json,
|
colorize_json,
|
||||||
|
|
@ -66,10 +59,6 @@ class SymbolNotFound(BrokerError):
|
||||||
"Symbol not found by broker search"
|
"Symbol not found by broker search"
|
||||||
|
|
||||||
|
|
||||||
class MarketNotFound(SymbolNotFound):
|
|
||||||
"Mkt-pair not found by broker search"
|
|
||||||
|
|
||||||
|
|
||||||
# TODO: these should probably be moved to `.tsp/.data`?
|
# TODO: these should probably be moved to `.tsp/.data`?
|
||||||
class NoData(BrokerError):
|
class NoData(BrokerError):
|
||||||
'''
|
'''
|
||||||
|
|
@ -108,19 +97,6 @@ class DataThrottle(BrokerError):
|
||||||
'''
|
'''
|
||||||
# TODO: add in throttle metrics/feedback
|
# TODO: add in throttle metrics/feedback
|
||||||
|
|
||||||
class SchemaMismatch(BrokerError):
|
|
||||||
'''
|
|
||||||
Market `Pair` fields mismatch, likely due to provider API update.
|
|
||||||
|
|
||||||
'''
|
|
||||||
|
|
||||||
# auto-register all `BrokerError` subtypes for
|
|
||||||
# tractor IPC exc-marshalling.
|
|
||||||
reg_err_types([
|
|
||||||
BrokerError,
|
|
||||||
*BrokerError.__subclasses__(),
|
|
||||||
])
|
|
||||||
|
|
||||||
|
|
||||||
def resproc(
|
def resproc(
|
||||||
resp: httpx.Response,
|
resp: httpx.Response,
|
||||||
|
|
@ -147,45 +123,3 @@ def resproc(
|
||||||
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
||||||
|
|
||||||
return msg if return_json else resp
|
return msg if return_json else resp
|
||||||
|
|
||||||
|
|
||||||
def get_or_raise_on_pair_schema_mismatch(
|
|
||||||
pair_type: Type[Struct],
|
|
||||||
fields_data: dict,
|
|
||||||
provider_name: str,
|
|
||||||
api_url: str|None = None,
|
|
||||||
) -> Struct:
|
|
||||||
'''
|
|
||||||
Boilerplate helper around assset-`Pair` field schema mismatches,
|
|
||||||
normally due to provider API updates.
|
|
||||||
|
|
||||||
'''
|
|
||||||
try:
|
|
||||||
pair: Struct = pair_type(**fields_data)
|
|
||||||
return pair
|
|
||||||
except TypeError as err:
|
|
||||||
|
|
||||||
from tractor.devx.pformat import ppfmt
|
|
||||||
repr_data: str = ppfmt(fields_data)
|
|
||||||
report: str = (
|
|
||||||
f'Field mismatch we need to codify!\n'
|
|
||||||
f'\n'
|
|
||||||
f'{pair_type!r}({repr_data})'
|
|
||||||
f'\n'
|
|
||||||
f'^^^ {err.args[0]!r} ^^^\n'
|
|
||||||
f'\n'
|
|
||||||
f"Don't panic, prolly {provider_name!r} "
|
|
||||||
f"changed their symbology schema..\n"
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
api_url
|
|
||||||
or
|
|
||||||
(api_url := pair_type._api_url)
|
|
||||||
):
|
|
||||||
report += (
|
|
||||||
f'\n'
|
|
||||||
f'Check out their API docs here:\n'
|
|
||||||
f'{api_url}\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
raise SchemaMismatch(report) from err
|
|
||||||
|
|
|
||||||
|
|
@ -52,25 +52,9 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# per-daemon-kind (sub)mod groups: declares which of our
|
# `brokerd` modules
|
||||||
# submods host the eps run by each daemon-actor kind as
|
__enable_modules__: list[str] = [
|
||||||
# defined by `piker.data.validate._eps`.
|
|
||||||
# NOTE: `get_mkt_info` and `open_symbol_search` both live
|
|
||||||
# in `.feed` for this backend (no `symbols.py`).
|
|
||||||
_brokerd_mods: list[str] = [
|
|
||||||
'api',
|
|
||||||
'broker',
|
|
||||||
]
|
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
|
||||||
'api',
|
'api',
|
||||||
'feed',
|
'feed',
|
||||||
|
'broker',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
|
||||||
_brokerd_mods
|
|
||||||
+
|
|
||||||
_datad_mods
|
|
||||||
))
|
|
||||||
|
|
|
||||||
|
|
@ -49,9 +49,6 @@ from piker import config
|
||||||
from piker.clearing._messages import (
|
from piker.clearing._messages import (
|
||||||
Order,
|
Order,
|
||||||
)
|
)
|
||||||
from piker.brokers._util import (
|
|
||||||
get_or_raise_on_pair_schema_mismatch,
|
|
||||||
)
|
|
||||||
from piker.accounting import (
|
from piker.accounting import (
|
||||||
Asset,
|
Asset,
|
||||||
digits_to_dec,
|
digits_to_dec,
|
||||||
|
|
@ -373,12 +370,20 @@ class Client:
|
||||||
item['filters'] = filters
|
item['filters'] = filters
|
||||||
|
|
||||||
pair_type: Type = PAIRTYPES[venue]
|
pair_type: Type = PAIRTYPES[venue]
|
||||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
try:
|
||||||
pair_type=pair_type,
|
pair: Pair = pair_type(**item)
|
||||||
fields_data=item,
|
except Exception as e:
|
||||||
provider_name='binance',
|
e.add_note(
|
||||||
api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
f'\n'
|
||||||
|
f'New or removed field we need to codify!\n'
|
||||||
|
f'pair-type: {pair_type!r}\n'
|
||||||
|
f'\n'
|
||||||
|
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
|
||||||
|
f'Check out their API docs here:\n'
|
||||||
|
f'\n'
|
||||||
|
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
|
||||||
)
|
)
|
||||||
|
raise
|
||||||
pair_table[pair.symbol.upper()] = pair
|
pair_table[pair.symbol.upper()] = pair
|
||||||
|
|
||||||
# update an additional top-level-cross-venue-table
|
# update an additional top-level-cross-venue-table
|
||||||
|
|
@ -604,11 +609,7 @@ class Client:
|
||||||
start_time = binance_timestamp(start_dt)
|
start_time = binance_timestamp(start_dt)
|
||||||
end_time = binance_timestamp(end_dt)
|
end_time = binance_timestamp(end_dt)
|
||||||
|
|
||||||
import tractor
|
bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
|
||||||
with tractor.devx.maybe_open_crash_handler():
|
|
||||||
bs_pair: Pair = self._pairs[
|
|
||||||
mkt.bs_fqme.upper()
|
|
||||||
]
|
|
||||||
|
|
||||||
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
||||||
bars = await self._api(
|
bars = await self._api(
|
||||||
|
|
|
||||||
|
|
@ -48,7 +48,6 @@ import tractor
|
||||||
from piker.brokers import (
|
from piker.brokers import (
|
||||||
open_cached_client,
|
open_cached_client,
|
||||||
NoData,
|
NoData,
|
||||||
MarketNotFound,
|
|
||||||
)
|
)
|
||||||
from piker._cacheables import (
|
from piker._cacheables import (
|
||||||
async_lifo_cache,
|
async_lifo_cache,
|
||||||
|
|
@ -204,13 +203,9 @@ async def stream_messages(
|
||||||
yield 'trade', piker_quote
|
yield 'trade', piker_quote
|
||||||
|
|
||||||
|
|
||||||
def make_sub(
|
def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
||||||
pairs: list[str],
|
|
||||||
sub_name: str,
|
|
||||||
uid: int,
|
|
||||||
) -> dict[str, str]:
|
|
||||||
'''
|
'''
|
||||||
Create a request subscription packet `dict`.
|
Create a request subscription packet dict.
|
||||||
|
|
||||||
- spot:
|
- spot:
|
||||||
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
|
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
|
||||||
|
|
@ -306,10 +301,6 @@ async def get_mkt_info(
|
||||||
|
|
||||||
# uppercase since kraken bs_mktid is always upper
|
# uppercase since kraken bs_mktid is always upper
|
||||||
if 'binance' not in fqme.lower():
|
if 'binance' not in fqme.lower():
|
||||||
log.warning(
|
|
||||||
f'Missing `.<provider>` part in fqme ??\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
)
|
|
||||||
fqme += '.binance'
|
fqme += '.binance'
|
||||||
|
|
||||||
mkt_mode: str = ''
|
mkt_mode: str = ''
|
||||||
|
|
@ -324,24 +315,6 @@ async def get_mkt_info(
|
||||||
venue: str = venue.upper()
|
venue: str = venue.upper()
|
||||||
venue_lower: str = venue.lower()
|
venue_lower: str = venue.lower()
|
||||||
|
|
||||||
if not venue:
|
|
||||||
if expiry:
|
|
||||||
expiry = f'.{expiry}'
|
|
||||||
expected: str = (
|
|
||||||
f'{mkt_ep}'
|
|
||||||
f'.<venue>'
|
|
||||||
f'{expiry}'
|
|
||||||
f'.{broker}'
|
|
||||||
)
|
|
||||||
raise MarketNotFound(
|
|
||||||
f'Invalid or missing .<venue> part in fqme?\n'
|
|
||||||
f'\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
f'expected-form>> {expected}\n'
|
|
||||||
f'\n'
|
|
||||||
f'Maybe you are missing a ".spot." ?\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
# XXX TODO: we should change the usdtm_futes name to just
|
# XXX TODO: we should change the usdtm_futes name to just
|
||||||
# usdm_futes (dropping the tether part) since it turns out that
|
# usdm_futes (dropping the tether part) since it turns out that
|
||||||
# there are indeed USD-tokens OTHER THEN tether being used as
|
# there are indeed USD-tokens OTHER THEN tether being used as
|
||||||
|
|
@ -359,8 +332,7 @@ async def get_mkt_info(
|
||||||
# TODO: handle coinm futes which have a margin asset that
|
# TODO: handle coinm futes which have a margin asset that
|
||||||
# is some crypto token!
|
# is some crypto token!
|
||||||
# https://binance-docs.github.io/apidocs/delivery/en/#exchange-information
|
# https://binance-docs.github.io/apidocs/delivery/en/#exchange-information
|
||||||
or
|
or 'btc' in venue_lower
|
||||||
'btc' in venue_lower
|
|
||||||
):
|
):
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
@ -371,21 +343,16 @@ async def get_mkt_info(
|
||||||
|
|
||||||
if (
|
if (
|
||||||
venue
|
venue
|
||||||
and
|
and 'spot' not in venue_lower
|
||||||
'spot' not in venue_lower
|
|
||||||
|
|
||||||
# XXX: catch all in case user doesn't know which
|
# XXX: catch all in case user doesn't know which
|
||||||
# venue they want (usdtm vs. coinm) and we can choose
|
# venue they want (usdtm vs. coinm) and we can choose
|
||||||
# a default (via config?) once we support coin-m APIs.
|
# a default (via config?) once we support coin-m APIs.
|
||||||
or
|
or 'perp' in venue_lower
|
||||||
'perp' in venue_lower
|
|
||||||
):
|
):
|
||||||
if not mkt_mode:
|
if not mkt_mode:
|
||||||
mkt_mode: str = f'{venue_lower}_futes'
|
mkt_mode: str = f'{venue_lower}_futes'
|
||||||
|
|
||||||
# tracing
|
|
||||||
# await tractor.pause()
|
|
||||||
|
|
||||||
async with open_cached_client(
|
async with open_cached_client(
|
||||||
'binance',
|
'binance',
|
||||||
) as client:
|
) as client:
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,6 @@ Per market data-type definitions and schemas types.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from typing import (
|
from typing import (
|
||||||
ClassVar,
|
|
||||||
Literal,
|
Literal,
|
||||||
)
|
)
|
||||||
from decimal import Decimal
|
from decimal import Decimal
|
||||||
|
|
@ -204,8 +203,6 @@ class FutesPair(Pair):
|
||||||
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
||||||
ns_path: str = 'piker.brokers.binance:FutesPair'
|
ns_path: str = 'piker.brokers.binance:FutesPair'
|
||||||
|
|
||||||
_api_url: ClassVar[str] = 'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
|
|
||||||
|
|
||||||
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
||||||
# processing..
|
# processing..
|
||||||
@property
|
@property
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,7 @@ from piker.log import (
|
||||||
get_logger,
|
get_logger,
|
||||||
)
|
)
|
||||||
from ..service import (
|
from ..service import (
|
||||||
maybe_spawn_datad,
|
maybe_spawn_brokerd,
|
||||||
maybe_open_pikerd,
|
maybe_open_pikerd,
|
||||||
)
|
)
|
||||||
from ..brokers import (
|
from ..brokers import (
|
||||||
|
|
@ -187,7 +187,7 @@ def brokercheck(config, broker):
|
||||||
|
|
||||||
'''
|
'''
|
||||||
async def bcheck_main():
|
async def bcheck_main():
|
||||||
async with maybe_spawn_datad(broker) as portal:
|
async with maybe_spawn_brokerd(broker) as portal:
|
||||||
await portal.run(run_test, broker)
|
await portal.run(run_test, broker)
|
||||||
await portal.cancel_actor()
|
await portal.cancel_actor()
|
||||||
|
|
||||||
|
|
@ -317,7 +317,7 @@ def record(config, rate, name, dhost, filename):
|
||||||
return
|
return
|
||||||
|
|
||||||
async def main(tries):
|
async def main(tries):
|
||||||
async with maybe_spawn_datad(
|
async with maybe_spawn_brokerd(
|
||||||
tries=tries, loglevel=loglevel
|
tries=tries, loglevel=loglevel
|
||||||
) as portal:
|
) as portal:
|
||||||
# run app "main"
|
# run app "main"
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,7 @@ import trio
|
||||||
|
|
||||||
from piker.log import get_logger
|
from piker.log import get_logger
|
||||||
from . import get_brokermod
|
from . import get_brokermod
|
||||||
from ..service import maybe_spawn_datad
|
from ..service import maybe_spawn_brokerd
|
||||||
from . import open_cached_client
|
from . import open_cached_client
|
||||||
from ..accounting import MktPair
|
from ..accounting import MktPair
|
||||||
|
|
||||||
|
|
@ -172,7 +172,7 @@ async def symbol_search(
|
||||||
# await tractor.devx._debug.maybe_init_greenback()
|
# await tractor.devx._debug.maybe_init_greenback()
|
||||||
# tractor.pause_from_sync()
|
# tractor.pause_from_sync()
|
||||||
|
|
||||||
async with maybe_spawn_datad(
|
async with maybe_spawn_brokerd(
|
||||||
mod.name,
|
mod.name,
|
||||||
infect_asyncio=getattr(
|
infect_asyncio=getattr(
|
||||||
mod,
|
mod,
|
||||||
|
|
|
||||||
|
|
@ -425,7 +425,7 @@ class DataFeed:
|
||||||
async def stream_to_file(
|
async def stream_to_file(
|
||||||
watchlist_name: str,
|
watchlist_name: str,
|
||||||
filename: str,
|
filename: str,
|
||||||
portal: tractor.Portal,
|
portal: tractor._portal.Portal,
|
||||||
tickers: List[str],
|
tickers: List[str],
|
||||||
brokermod: ModuleType,
|
brokermod: ModuleType,
|
||||||
rate: int,
|
rate: int,
|
||||||
|
|
|
||||||
|
|
@ -47,25 +47,13 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# per-daemon-kind (sub)mod groups: declares which of our
|
# tractor RPC enable arg
|
||||||
# submods host the eps run by each daemon-actor kind as
|
__enable_modules__: list[str] = [
|
||||||
# defined by `piker.data.validate._eps`.
|
|
||||||
# NOTE: datad-only backend (no `broker.py` yet)!
|
|
||||||
_brokerd_mods: list[str] = []
|
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
|
||||||
'api',
|
'api',
|
||||||
'feed',
|
'feed',
|
||||||
|
# 'broker',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
|
||||||
_brokerd_mods
|
|
||||||
+
|
|
||||||
_datad_mods
|
|
||||||
))
|
|
||||||
|
|
||||||
# passed to ``tractor.ActorNursery.start_actor()``
|
# passed to ``tractor.ActorNursery.start_actor()``
|
||||||
_spawn_kwargs = {
|
_spawn_kwargs = {
|
||||||
'infect_asyncio': True,
|
'infect_asyncio': True,
|
||||||
|
|
|
||||||
|
|
@ -23,6 +23,7 @@ from contextlib import (
|
||||||
asynccontextmanager as acm,
|
asynccontextmanager as acm,
|
||||||
)
|
)
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
from functools import partial
|
||||||
import time
|
import time
|
||||||
from typing import (
|
from typing import (
|
||||||
Any,
|
Any,
|
||||||
|
|
@ -37,7 +38,7 @@ from rapidfuzz import process as fuzzy
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from tractor.trionics import (
|
from tractor.trionics import (
|
||||||
broadcast_receiver,
|
broadcast_receiver,
|
||||||
maybe_open_context,
|
maybe_open_context
|
||||||
collapse_eg,
|
collapse_eg,
|
||||||
)
|
)
|
||||||
from tractor import to_asyncio
|
from tractor import to_asyncio
|
||||||
|
|
@ -523,12 +524,13 @@ async def maybe_open_feed_handler() -> trio.abc.ReceiveStream:
|
||||||
|
|
||||||
|
|
||||||
async def aio_price_feed_relay(
|
async def aio_price_feed_relay(
|
||||||
chan: to_asyncio.LinkedTaskChannel,
|
|
||||||
fh: FeedHandler,
|
fh: FeedHandler,
|
||||||
instrument: Symbol,
|
instrument: Symbol,
|
||||||
|
from_trio: asyncio.Queue,
|
||||||
|
to_trio: trio.abc.SendChannel,
|
||||||
) -> None:
|
) -> None:
|
||||||
async def _trade(data: dict, receipt_timestamp):
|
async def _trade(data: dict, receipt_timestamp):
|
||||||
chan.send_nowait(('trade', {
|
to_trio.send_nowait(('trade', {
|
||||||
'symbol': cb_sym_to_deribit_inst(
|
'symbol': cb_sym_to_deribit_inst(
|
||||||
str_to_cb_sym(data.symbol)).lower(),
|
str_to_cb_sym(data.symbol)).lower(),
|
||||||
'last': data,
|
'last': data,
|
||||||
|
|
@ -538,7 +540,7 @@ async def aio_price_feed_relay(
|
||||||
}))
|
}))
|
||||||
|
|
||||||
async def _l1(data: dict, receipt_timestamp):
|
async def _l1(data: dict, receipt_timestamp):
|
||||||
chan.send_nowait(('l1', {
|
to_trio.send_nowait(('l1', {
|
||||||
'symbol': cb_sym_to_deribit_inst(
|
'symbol': cb_sym_to_deribit_inst(
|
||||||
str_to_cb_sym(data.symbol)).lower(),
|
str_to_cb_sym(data.symbol)).lower(),
|
||||||
'ticks': [
|
'ticks': [
|
||||||
|
|
@ -568,7 +570,7 @@ async def aio_price_feed_relay(
|
||||||
install_signal_handlers=False)
|
install_signal_handlers=False)
|
||||||
|
|
||||||
# sync with trio
|
# sync with trio
|
||||||
chan.started_nowait(None)
|
to_trio.send_nowait(None)
|
||||||
|
|
||||||
await asyncio.sleep(float('inf'))
|
await asyncio.sleep(float('inf'))
|
||||||
|
|
||||||
|
|
@ -579,9 +581,11 @@ async def open_price_feed(
|
||||||
) -> trio.abc.ReceiveStream:
|
) -> trio.abc.ReceiveStream:
|
||||||
async with maybe_open_feed_handler() as fh:
|
async with maybe_open_feed_handler() as fh:
|
||||||
async with to_asyncio.open_channel_from(
|
async with to_asyncio.open_channel_from(
|
||||||
|
partial(
|
||||||
aio_price_feed_relay,
|
aio_price_feed_relay,
|
||||||
fh=fh,
|
fh,
|
||||||
instrument=instrument,
|
instrument
|
||||||
|
)
|
||||||
) as (chan, first):
|
) as (chan, first):
|
||||||
yield chan
|
yield chan
|
||||||
|
|
||||||
|
|
@ -607,9 +611,10 @@ async def maybe_open_price_feed(
|
||||||
|
|
||||||
|
|
||||||
async def aio_order_feed_relay(
|
async def aio_order_feed_relay(
|
||||||
chan: to_asyncio.LinkedTaskChannel,
|
|
||||||
fh: FeedHandler,
|
fh: FeedHandler,
|
||||||
instrument: Symbol,
|
instrument: Symbol,
|
||||||
|
from_trio: asyncio.Queue,
|
||||||
|
to_trio: trio.abc.SendChannel,
|
||||||
) -> None:
|
) -> None:
|
||||||
async def _fill(data: dict, receipt_timestamp):
|
async def _fill(data: dict, receipt_timestamp):
|
||||||
breakpoint()
|
breakpoint()
|
||||||
|
|
@ -632,7 +637,7 @@ async def aio_order_feed_relay(
|
||||||
install_signal_handlers=False)
|
install_signal_handlers=False)
|
||||||
|
|
||||||
# sync with trio
|
# sync with trio
|
||||||
chan.started_nowait(None)
|
to_trio.send_nowait(None)
|
||||||
|
|
||||||
await asyncio.sleep(float('inf'))
|
await asyncio.sleep(float('inf'))
|
||||||
|
|
||||||
|
|
@ -643,9 +648,11 @@ async def open_order_feed(
|
||||||
) -> trio.abc.ReceiveStream:
|
) -> trio.abc.ReceiveStream:
|
||||||
async with maybe_open_feed_handler() as fh:
|
async with maybe_open_feed_handler() as fh:
|
||||||
async with to_asyncio.open_channel_from(
|
async with to_asyncio.open_channel_from(
|
||||||
|
partial(
|
||||||
aio_order_feed_relay,
|
aio_order_feed_relay,
|
||||||
fh=fh,
|
fh,
|
||||||
instrument=instrument,
|
instrument
|
||||||
|
)
|
||||||
) as (chan, first):
|
) as (chan, first):
|
||||||
yield chan
|
yield chan
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -65,18 +65,17 @@ _brokerd_mods: list[str] = [
|
||||||
]
|
]
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
_datad_mods: list[str] = [
|
||||||
'api',
|
|
||||||
'feed',
|
'feed',
|
||||||
'symbols',
|
'symbols',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
# tractor RPC enable arg
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
__enable_modules__: list[str] = (
|
||||||
_brokerd_mods
|
_brokerd_mods
|
||||||
+
|
+
|
||||||
_datad_mods
|
_datad_mods
|
||||||
))
|
)
|
||||||
|
|
||||||
# passed to ``tractor.ActorNursery.start_actor()``
|
# passed to ``tractor.ActorNursery.start_actor()``
|
||||||
_spawn_kwargs = {
|
_spawn_kwargs = {
|
||||||
|
|
|
||||||
|
|
@ -29,10 +29,7 @@ from dataclasses import (
|
||||||
asdict,
|
asdict,
|
||||||
astuple,
|
astuple,
|
||||||
)
|
)
|
||||||
from datetime import (
|
from datetime import datetime
|
||||||
UTC,
|
|
||||||
datetime,
|
|
||||||
)
|
|
||||||
from functools import (
|
from functools import (
|
||||||
partial,
|
partial,
|
||||||
)
|
)
|
||||||
|
|
@ -98,7 +95,6 @@ from .symbols import (
|
||||||
)
|
)
|
||||||
from ...log import get_logger
|
from ...log import get_logger
|
||||||
from .venues import (
|
from .venues import (
|
||||||
is_expired,
|
|
||||||
is_venue_open,
|
is_venue_open,
|
||||||
sesh_times,
|
sesh_times,
|
||||||
is_venue_closure,
|
is_venue_closure,
|
||||||
|
|
@ -398,13 +394,15 @@ class Client:
|
||||||
default_dt_duration
|
default_dt_duration
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# TODO: maybe remove all this?
|
||||||
global _enters
|
global _enters
|
||||||
if end_dt is None:
|
if end_dt is None:
|
||||||
end_date_time: str = ''
|
end_dt: str = ''
|
||||||
|
|
||||||
else:
|
else:
|
||||||
end_date_time = end_dt.astimezone(UTC).strftime(
|
est_end_dt = end_dt.in_tz('EST')
|
||||||
'%Y%m%d-%H:%M:%S'
|
if est_end_dt != end_dt:
|
||||||
)
|
breakpoint()
|
||||||
|
|
||||||
_enters += 1
|
_enters += 1
|
||||||
|
|
||||||
|
|
@ -413,7 +411,7 @@ class Client:
|
||||||
|
|
||||||
kwargs: dict[str, Any] = dict(
|
kwargs: dict[str, Any] = dict(
|
||||||
contract=contract,
|
contract=contract,
|
||||||
endDateTime=end_date_time,
|
endDateTime=end_dt,
|
||||||
formatDate=2,
|
formatDate=2,
|
||||||
|
|
||||||
# OHLC sampling values:
|
# OHLC sampling values:
|
||||||
|
|
@ -498,7 +496,7 @@ class Client:
|
||||||
await self.ib.reqContractDetailsAsync(contract)
|
await self.ib.reqContractDetailsAsync(contract)
|
||||||
)[0]
|
)[0]
|
||||||
# convert to makt-native tz
|
# convert to makt-native tz
|
||||||
tz: str = details.timeZoneId or 'EST'
|
tz: str = details.timeZoneId
|
||||||
end_dt = end_dt.in_tz(tz)
|
end_dt = end_dt.in_tz(tz)
|
||||||
first_dt: DateTime = from_timestamp(first).in_tz(tz)
|
first_dt: DateTime = from_timestamp(first).in_tz(tz)
|
||||||
last_dt: DateTime = from_timestamp(last).in_tz(tz)
|
last_dt: DateTime = from_timestamp(last).in_tz(tz)
|
||||||
|
|
@ -510,18 +508,10 @@ class Client:
|
||||||
_open_now: bool = is_venue_open(
|
_open_now: bool = is_venue_open(
|
||||||
con_deats=details,
|
con_deats=details,
|
||||||
)
|
)
|
||||||
_is_expired: bool = is_expired(
|
|
||||||
con_deats=details,
|
|
||||||
)
|
|
||||||
|
|
||||||
# XXX, do gap detections.
|
# XXX, do gap detections.
|
||||||
has_closure_gap: bool = False
|
has_closure_gap: bool = False
|
||||||
if (
|
if (
|
||||||
# XXX, expired tracts can't be introspected
|
|
||||||
# for open/closure intervals due to ib's chitty
|
|
||||||
# details seemingly..
|
|
||||||
not _is_expired
|
|
||||||
and
|
|
||||||
last_dt.add(seconds=sample_period_s)
|
last_dt.add(seconds=sample_period_s)
|
||||||
<
|
<
|
||||||
end_dt
|
end_dt
|
||||||
|
|
@ -968,26 +958,8 @@ class Client:
|
||||||
|
|
||||||
else:
|
else:
|
||||||
raise
|
raise
|
||||||
# XXX: ambiguous/unqualifiable contracts are
|
|
||||||
# returned as `None` entries by
|
|
||||||
# `qualifyContractsAsync()` (which also logs an
|
|
||||||
# "Ambiguous contract" warning listing the
|
|
||||||
# possible matches) so filter them BEFORE any
|
|
||||||
# attr access B)
|
|
||||||
contracts: list[Contract] = [
|
|
||||||
tract
|
|
||||||
for tract in contracts
|
|
||||||
if tract is not None
|
|
||||||
]
|
|
||||||
if not contracts:
|
if not contracts:
|
||||||
raise ValueError(
|
raise ValueError(f"No contract could be found {con}")
|
||||||
f'No (unambiguous) contract could be '
|
|
||||||
f'qualified for {con!r}\n'
|
|
||||||
f'\n'
|
|
||||||
f'If a stonk, you likely need a (more) '
|
|
||||||
f'venue-qualified fqme,\n'
|
|
||||||
f"eg. 'gld.arca.ib' instead of 'gld.ib'\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
# pack all contracts into cache
|
# pack all contracts into cache
|
||||||
for tract in contracts:
|
for tract in contracts:
|
||||||
|
|
@ -1165,16 +1137,10 @@ class Client:
|
||||||
try:
|
try:
|
||||||
con: Contract = self._contracts[symbol]
|
con: Contract = self._contracts[symbol]
|
||||||
except KeyError:
|
except KeyError:
|
||||||
# require that the mkt's contract has been previously
|
# require that the symbol has been previously cached by
|
||||||
# qualified and cached (see `.symbols.get_mkt_info()`
|
# a data feed request - ensure we aren't making orders
|
||||||
# which is run for any feed-init OR lazily by the
|
# against non-known prices.
|
||||||
# order-request handler in `.broker`) - ensures we
|
raise RuntimeError("Can not order {symbol}, no live feed?")
|
||||||
# aren't making orders against unknown contracts.
|
|
||||||
raise RuntimeError(
|
|
||||||
f'Can not order {symbol!r}, '
|
|
||||||
f'no qualified contract cached!?\n'
|
|
||||||
f'_contracts: {list(self._contracts)!r}\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
trade = self.ib.placeOrder(
|
trade = self.ib.placeOrder(
|
||||||
|
|
@ -1365,21 +1331,6 @@ async def load_aio_clients(
|
||||||
ib = None
|
ib = None
|
||||||
client = None
|
client = None
|
||||||
|
|
||||||
# XXX: post (brokerd vs. datad)-split BOTH per-broker
|
|
||||||
# daemons connect to the same API gw/tws endpoint(s); to
|
|
||||||
# avoid `clientId` collisions (and the long conn-timeout
|
|
||||||
# retry cycle they cause) we offset the data-daemon's
|
|
||||||
# default id-range to be disjoint from `brokerd.ib`'s
|
|
||||||
# (which also retries with `client_id + i` increments).
|
|
||||||
if client_id == 6116: # the default from above
|
|
||||||
aname: str = tractor.current_actor().name
|
|
||||||
if 'datad' in aname:
|
|
||||||
client_id += 16
|
|
||||||
# ad-hoc (test/cli) actors get their own range to
|
|
||||||
# avoid clashing with any live daemon-tree's conns.
|
|
||||||
elif 'brokerd' not in aname:
|
|
||||||
client_id += 32
|
|
||||||
|
|
||||||
# attempt to get connection info from config; if no .toml entry
|
# attempt to get connection info from config; if no .toml entry
|
||||||
# exists, we try to load from a default localhost connection.
|
# exists, we try to load from a default localhost connection.
|
||||||
localhost = '127.0.0.1'
|
localhost = '127.0.0.1'
|
||||||
|
|
@ -1633,17 +1584,6 @@ class MethodProxy:
|
||||||
self.event_table = event_table
|
self.event_table = event_table
|
||||||
self._aio_ns = asyncio_ns
|
self._aio_ns = asyncio_ns
|
||||||
|
|
||||||
# request-id counter for correlating each method
|
|
||||||
# call to its (eventual) response; necessary since
|
|
||||||
# the (single, shared) chan has NO other ordering
|
|
||||||
# guarantee whenever a caller task is CANCELLED
|
|
||||||
# (eg. by a search-req timeout) after sending its
|
|
||||||
# request but before consuming the response: the
|
|
||||||
# "orphaned" response otherwise gets mis-delivered
|
|
||||||
# to the next caller causing an off-by-one skew of
|
|
||||||
# every result thereafter!
|
|
||||||
self._mids = itertools.count()
|
|
||||||
|
|
||||||
async def _run_method(
|
async def _run_method(
|
||||||
self,
|
self,
|
||||||
*,
|
*,
|
||||||
|
|
@ -1657,10 +1597,10 @@ class MethodProxy:
|
||||||
|
|
||||||
'''
|
'''
|
||||||
chan = self.chan
|
chan = self.chan
|
||||||
mid: int = next(self._mids)
|
await chan.send((meth, kwargs))
|
||||||
await chan.send((meth, kwargs, mid))
|
|
||||||
|
|
||||||
while not chan.closed():
|
while not chan.closed():
|
||||||
|
# send through method + ``kwargs: dict`` as pair
|
||||||
msg = await chan.receive()
|
msg = await chan.receive()
|
||||||
|
|
||||||
# TODO: implement reconnect functionality like
|
# TODO: implement reconnect functionality like
|
||||||
|
|
@ -1670,50 +1610,23 @@ class MethodProxy:
|
||||||
# except ConnectionError:
|
# except ConnectionError:
|
||||||
# self.reset()
|
# self.reset()
|
||||||
|
|
||||||
match msg:
|
# print(f'NEXT MSG: {msg}')
|
||||||
# OUR method-call response B)
|
|
||||||
case {'mid': resp_mid, 'result': res} if (
|
# TODO: py3.10 ``match:`` syntax B)
|
||||||
resp_mid == mid
|
if 'result' in msg:
|
||||||
):
|
res = msg.get('result')
|
||||||
return res
|
return res
|
||||||
|
|
||||||
case {'mid': resp_mid, 'exception': err} if (
|
elif 'exception' in msg:
|
||||||
resp_mid == mid
|
err = msg.get('exception')
|
||||||
):
|
|
||||||
raise err
|
raise err
|
||||||
|
|
||||||
# an "orphaned" response to some prior
|
elif 'error' in msg:
|
||||||
# (cancelled) caller's request; drop it and
|
etype, emsg = msg
|
||||||
# keep waiting for ours.
|
|
||||||
case {'mid': resp_mid}:
|
|
||||||
log.warning(
|
|
||||||
f'Dropping stale method-resp,\n'
|
|
||||||
f'mid: {resp_mid} (ours: {mid})\n'
|
|
||||||
f'(a prior caller prolly got '
|
|
||||||
f'cancelled before its resp?)\n'
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# out-of-band (inline) client error: raise
|
|
||||||
# to the current caller as before.
|
|
||||||
case {'exception': err}:
|
|
||||||
raise err
|
|
||||||
|
|
||||||
case ('error', emsg):
|
|
||||||
log.warning(f'IB error relay: {emsg}')
|
log.warning(f'IB error relay: {emsg}')
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# routine (api-farm conn) status events
|
else:
|
||||||
# relayed inline by `Client.inline_errors()`;
|
|
||||||
# not a response to our method call so just
|
|
||||||
# log at info and keep waiting.
|
|
||||||
case ('event', emsg):
|
|
||||||
log.info(
|
|
||||||
f'IB status event relay: {emsg}'
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
case _:
|
|
||||||
log.warning(f'UNKNOWN IB MSG: {msg}')
|
log.warning(f'UNKNOWN IB MSG: {msg}')
|
||||||
|
|
||||||
def status_event(
|
def status_event(
|
||||||
|
|
@ -1767,31 +1680,22 @@ async def open_aio_client_method_relay(
|
||||||
log.info('asyncio `Client` method-proxy SHUTDOWN!')
|
log.info('asyncio `Client` method-proxy SHUTDOWN!')
|
||||||
break
|
break
|
||||||
|
|
||||||
case (meth_name, kwargs, mid):
|
case (meth_name, kwargs):
|
||||||
|
meth_name, kwargs = msg
|
||||||
meth = getattr(client, meth_name)
|
meth = getattr(client, meth_name)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
resp = await meth(**kwargs)
|
resp = await meth(**kwargs)
|
||||||
# echo the msg back tagged with the
|
# echo the msg back
|
||||||
# req-id for caller-side correlation.
|
chan.send_nowait({'result': resp})
|
||||||
chan.send_nowait({
|
|
||||||
'mid': mid,
|
|
||||||
'result': resp,
|
|
||||||
})
|
|
||||||
|
|
||||||
# XXX: relay ALL (non-cancel) errors to the
|
except (
|
||||||
# `trio`-side caller (which re-raises in the
|
RequestError,
|
||||||
# `MethodProxy._run_method()` frame) instead
|
|
||||||
# of crashing this relay task and thus the
|
# TODO: relay all errors to trio?
|
||||||
# whole proxy/dialog; critical post the
|
# BaseException,
|
||||||
# (datad|brokerd)-split where eg. contract
|
) as err:
|
||||||
# qualification failures are expected to be
|
chan.send_nowait({'exception': err})
|
||||||
# caught per-request by order/warmup code!
|
|
||||||
except Exception as err:
|
|
||||||
chan.send_nowait({
|
|
||||||
'mid': mid,
|
|
||||||
'exception': err,
|
|
||||||
})
|
|
||||||
|
|
||||||
case {'error': content}:
|
case {'error': content}:
|
||||||
chan.send_nowait({'exception': content})
|
chan.send_nowait({'exception': content})
|
||||||
|
|
|
||||||
|
|
@ -89,9 +89,8 @@ from .api import (
|
||||||
MethodProxy,
|
MethodProxy,
|
||||||
)
|
)
|
||||||
from .symbols import (
|
from .symbols import (
|
||||||
cache_contract,
|
|
||||||
con2fqme,
|
con2fqme,
|
||||||
get_mkt_info,
|
# get_mkt_info,
|
||||||
)
|
)
|
||||||
from .ledger import (
|
from .ledger import (
|
||||||
norm_trade_records,
|
norm_trade_records,
|
||||||
|
|
@ -139,7 +138,6 @@ async def handle_order_requests(
|
||||||
ems_order_stream: tractor.MsgStream,
|
ems_order_stream: tractor.MsgStream,
|
||||||
accounts_def: dict[str, str],
|
accounts_def: dict[str, str],
|
||||||
flows: OrderDialogs,
|
flows: OrderDialogs,
|
||||||
proxies: dict[str, MethodProxy],
|
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
||||||
|
|
@ -182,58 +180,6 @@ async def handle_order_requests(
|
||||||
if action in {'buy', 'sell'}:
|
if action in {'buy', 'sell'}:
|
||||||
# validate
|
# validate
|
||||||
order = BrokerdOrder(**request_msg)
|
order = BrokerdOrder(**request_msg)
|
||||||
fqme: str = order.symbol
|
|
||||||
|
|
||||||
# XXX: lazily qualify and cache the contract for
|
|
||||||
# this mkt since, post the (datad|brokerd)-split,
|
|
||||||
# this trading-only actor will NOT have had its
|
|
||||||
# (api) `Client._contracts` pre-warmed by any
|
|
||||||
# in-proc feed setup (which now runs in the
|
|
||||||
# `datad.ib` sibling); we run the SAME symbology
|
|
||||||
# resolution ep the feed-side uses so the cache
|
|
||||||
# key (`MktPair.bs_fqme`) matches what
|
|
||||||
# `Client.submit_limit()` looks up.
|
|
||||||
# NOTE: normally a no-op since
|
|
||||||
# `open_trade_dialog()` eagerly pre-qualifies all
|
|
||||||
# already-open pp/order mkts at startup; this
|
|
||||||
# only fires for a brand-new (to this daemon)
|
|
||||||
# mkt's first order.
|
|
||||||
if fqme not in client._contracts:
|
|
||||||
proxy: MethodProxy = proxies[account]
|
|
||||||
try:
|
|
||||||
(
|
|
||||||
mkt,
|
|
||||||
details,
|
|
||||||
) = await get_mkt_info(
|
|
||||||
fqme,
|
|
||||||
proxy=proxy,
|
|
||||||
)
|
|
||||||
# XXX: explicit write since the lifo-cache
|
|
||||||
# may deliver a hit (body skipped) when
|
|
||||||
# another acct/client already qualified
|
|
||||||
# this fqme.
|
|
||||||
cache_contract(
|
|
||||||
client,
|
|
||||||
mkt,
|
|
||||||
details.contract,
|
|
||||||
)
|
|
||||||
except Exception as err:
|
|
||||||
log.exception(
|
|
||||||
f'Failed to qualify contract for\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
)
|
|
||||||
await ems_order_stream.send(
|
|
||||||
BrokerdError(
|
|
||||||
oid=oid,
|
|
||||||
symbol=fqme,
|
|
||||||
reason=(
|
|
||||||
f'No contract could be '
|
|
||||||
f'qualified for {fqme!r}:\n'
|
|
||||||
f'{err!r}'
|
|
||||||
),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# XXX: by default 0 tells ``ib_async`` methods that
|
# XXX: by default 0 tells ``ib_async`` methods that
|
||||||
# there is no existing order so ask the client to create
|
# there is no existing order so ask the client to create
|
||||||
|
|
@ -245,34 +191,15 @@ async def handle_order_requests(
|
||||||
reqid = int(reqid)
|
reqid = int(reqid)
|
||||||
|
|
||||||
# call our client api to submit the order
|
# call our client api to submit the order
|
||||||
# NOTE: guard with order-error relay (vs. crashing
|
|
||||||
# this dialog and thus ALL order ctl for the
|
|
||||||
# backend) so one bad submission can't take down
|
|
||||||
# the daemon's clearing loop.
|
|
||||||
try:
|
|
||||||
reqid = client.submit_limit(
|
reqid = client.submit_limit(
|
||||||
oid=order.oid,
|
oid=order.oid,
|
||||||
symbol=fqme,
|
symbol=order.symbol,
|
||||||
price=order.price,
|
price=order.price,
|
||||||
action=order.action,
|
action=order.action,
|
||||||
size=order.size,
|
size=order.size,
|
||||||
account=acct_number,
|
account=acct_number,
|
||||||
reqid=reqid,
|
reqid=reqid,
|
||||||
)
|
)
|
||||||
except Exception as err:
|
|
||||||
log.exception(
|
|
||||||
f'Order submission failed for\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
)
|
|
||||||
await ems_order_stream.send(
|
|
||||||
BrokerdError(
|
|
||||||
oid=oid,
|
|
||||||
symbol=fqme,
|
|
||||||
reason=f'Submission error: {err!r}',
|
|
||||||
)
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
str_reqid: str = str(reqid)
|
str_reqid: str = str(reqid)
|
||||||
if reqid is None:
|
if reqid is None:
|
||||||
err_msg = BrokerdError(
|
err_msg = BrokerdError(
|
||||||
|
|
@ -304,21 +231,20 @@ async def handle_order_requests(
|
||||||
|
|
||||||
async def recv_trade_updates(
|
async def recv_trade_updates(
|
||||||
|
|
||||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
|
||||||
client: Client,
|
client: Client,
|
||||||
|
to_trio: trio.abc.SendChannel,
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Receive and relay order control and positioning
|
Receive and relay order control and positioning related events
|
||||||
related events from `ib_async`, pack as tuples and
|
from `ib_async`, pack as tuples and push over mem-chan to our
|
||||||
push over mem-chan to our trio relay task for
|
trio relay task for processing and relay to EMS.
|
||||||
processing and relay to EMS.
|
|
||||||
|
|
||||||
'''
|
'''
|
||||||
client.inline_errors(chan)
|
client.inline_errors(to_trio)
|
||||||
|
|
||||||
# sync with trio task
|
# sync with trio task
|
||||||
chan.started_nowait(client.ib)
|
to_trio.send_nowait(client.ib)
|
||||||
|
|
||||||
def push_tradesies(
|
def push_tradesies(
|
||||||
eventkit_obj,
|
eventkit_obj,
|
||||||
|
|
@ -356,7 +282,7 @@ async def recv_trade_updates(
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# emit event name + relevant ibis internal objects
|
# emit event name + relevant ibis internal objects
|
||||||
chan.send_nowait((event_name, emit))
|
to_trio.send_nowait((event_name, emit))
|
||||||
except trio.BrokenResourceError:
|
except trio.BrokenResourceError:
|
||||||
log.exception(f'Disconnected from {eventkit_obj} updates')
|
log.exception(f'Disconnected from {eventkit_obj} updates')
|
||||||
eventkit_obj.disconnect(push_tradesies)
|
eventkit_obj.disconnect(push_tradesies)
|
||||||
|
|
@ -621,14 +547,10 @@ async def open_trade_dialog(
|
||||||
|
|
||||||
) -> AsyncIterator[dict[str, Any]]:
|
) -> AsyncIterator[dict[str, Any]]:
|
||||||
|
|
||||||
# XXX: ONLY adjust the level of this (sub)mod's logger;
|
get_console_log(
|
||||||
# attaching a (stderr) handler (via `get_console_log()`)
|
level=loglevel,
|
||||||
# here would DOUBLE-print every record since the daemon
|
name=__name__,
|
||||||
# fixture (`.._daemon._setup_persistent_brokerd()`)
|
)
|
||||||
# already enables the console handler on the parent
|
|
||||||
# subsys logger which all our records propagate to!
|
|
||||||
if loglevel:
|
|
||||||
log.setLevel(loglevel.upper())
|
|
||||||
|
|
||||||
# task local msg dialog tracking
|
# task local msg dialog tracking
|
||||||
flows = OrderDialogs()
|
flows = OrderDialogs()
|
||||||
|
|
@ -865,64 +787,6 @@ async def open_trade_dialog(
|
||||||
for msg in order_msgs:
|
for msg in order_msgs:
|
||||||
await ems_stream.send(msg)
|
await ems_stream.send(msg)
|
||||||
|
|
||||||
# XXX: eagerly pre-qualify (and cache) the
|
|
||||||
# contracts for all already-open pps and
|
|
||||||
# orders so that (live) order submission
|
|
||||||
# NEVER pays a first-request qualification
|
|
||||||
# delay; any brand-new mkt is still lazily
|
|
||||||
# qualified by `handle_order_requests()` on
|
|
||||||
# its first submission. NOTE: since this
|
|
||||||
# runs BEFORE the order handler task is
|
|
||||||
# even started, no order can clear until
|
|
||||||
# the warmup completes (early reqs just
|
|
||||||
# buffer in the ems IPC stream) B)
|
|
||||||
warmup_fqmes: set[str] = {
|
|
||||||
msg.symbol
|
|
||||||
for msg in all_positions
|
|
||||||
}
|
|
||||||
warmup_fqmes.update(
|
|
||||||
msg.req.symbol
|
|
||||||
for msg in order_msgs
|
|
||||||
)
|
|
||||||
unique_clients: set[Client] = set(
|
|
||||||
aioclients.values()
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
warmup_fqmes
|
|
||||||
and
|
|
||||||
proxies
|
|
||||||
):
|
|
||||||
a_proxy: MethodProxy = next(
|
|
||||||
iter(proxies.values())
|
|
||||||
)
|
|
||||||
for fqme in warmup_fqmes:
|
|
||||||
try:
|
|
||||||
(
|
|
||||||
mkt,
|
|
||||||
details,
|
|
||||||
) = await get_mkt_info(
|
|
||||||
fqme,
|
|
||||||
proxy=a_proxy,
|
|
||||||
)
|
|
||||||
except Exception as err:
|
|
||||||
# XXX: non-fatal; an
|
|
||||||
# un-warmed mkt just falls
|
|
||||||
# back to the lazy qualify
|
|
||||||
# in the order handler.
|
|
||||||
log.warning(
|
|
||||||
f'Failed to pre-qualify\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
f'err: {err!r}\n'
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
for _client in unique_clients:
|
|
||||||
cache_contract(
|
|
||||||
_client,
|
|
||||||
mkt,
|
|
||||||
details.contract,
|
|
||||||
)
|
|
||||||
|
|
||||||
for client in set(aioclients.values()):
|
for client in set(aioclients.values()):
|
||||||
trade_event_stream: LinkedTaskChannel = await tn.start(
|
trade_event_stream: LinkedTaskChannel = await tn.start(
|
||||||
open_trade_event_stream,
|
open_trade_event_stream,
|
||||||
|
|
@ -936,7 +800,6 @@ async def open_trade_dialog(
|
||||||
ems_stream,
|
ems_stream,
|
||||||
accounts_def,
|
accounts_def,
|
||||||
flows,
|
flows,
|
||||||
proxies,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# allocate event relay tasks for each client connection
|
# allocate event relay tasks for each client connection
|
||||||
|
|
@ -1299,9 +1162,9 @@ async def deliver_trade_events(
|
||||||
case 'execDetailsEvent':
|
case 'execDetailsEvent':
|
||||||
# unpack attrs pep-0526 style.
|
# unpack attrs pep-0526 style.
|
||||||
trade: Trade
|
trade: Trade
|
||||||
|
con: Contract = trade.contract
|
||||||
fill: Fill
|
fill: Fill
|
||||||
trade, fill = item
|
trade, fill = item
|
||||||
con: Contract = trade.contract
|
|
||||||
execu: Execution = fill.execution
|
execu: Execution = fill.execution
|
||||||
execid: str = execu.execId
|
execid: str = execu.execId
|
||||||
report: CommissionReport = fill.commissionReport
|
report: CommissionReport = fill.commissionReport
|
||||||
|
|
@ -1443,15 +1306,7 @@ async def deliver_trade_events(
|
||||||
elif isinstance(err, str):
|
elif isinstance(err, str):
|
||||||
code_part, _, reason = err.rpartition(']')
|
code_part, _, reason = err.rpartition(']')
|
||||||
if code_part:
|
if code_part:
|
||||||
for prefix_patt in [
|
_, _, code = code_part.partition('[code')
|
||||||
'[Errno ',
|
|
||||||
'[code ',
|
|
||||||
]:
|
|
||||||
code_part, _, code = code_part.partition()
|
|
||||||
if code:
|
|
||||||
code = int(code)
|
|
||||||
break
|
|
||||||
|
|
||||||
reqid: str = '<unknown>'
|
reqid: str = '<unknown>'
|
||||||
|
|
||||||
# "Warning:" msg codes,
|
# "Warning:" msg codes,
|
||||||
|
|
@ -1470,8 +1325,6 @@ async def deliver_trade_events(
|
||||||
|
|
||||||
# XXX known special (ignore) cases
|
# XXX known special (ignore) cases
|
||||||
elif code in {
|
elif code in {
|
||||||
# ^TODO, if this is it we should definitely raise
|
|
||||||
# or at least provide for reconnect attempts?
|
|
||||||
200, # uhh.. ni idea
|
200, # uhh.. ni idea
|
||||||
|
|
||||||
# hist pacing / connectivity
|
# hist pacing / connectivity
|
||||||
|
|
@ -1482,18 +1335,10 @@ async def deliver_trade_events(
|
||||||
# 'No market data during competing live session'
|
# 'No market data during competing live session'
|
||||||
1669,
|
1669,
|
||||||
}:
|
}:
|
||||||
pcc: str = "Peer closed connection."
|
|
||||||
if pcc in err:
|
|
||||||
# TODO, emit and raise?
|
|
||||||
# [ ] try for reconnect?
|
|
||||||
# raise BrokerdError(
|
|
||||||
await tractor.pause()
|
|
||||||
|
|
||||||
log.error(
|
log.error(
|
||||||
f'Order-API-error which is non-cancel-causing ?!\n'
|
f'Order-API-error which is non-cancel-causing ?!\n'
|
||||||
f'\n'
|
f'\n'
|
||||||
f'code={code!r}\n'
|
f'{pformat(err)}\n'
|
||||||
f'err={pformat(err)}\n'
|
|
||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -32,6 +32,7 @@ import time
|
||||||
from typing import (
|
from typing import (
|
||||||
Any,
|
Any,
|
||||||
Callable,
|
Callable,
|
||||||
|
TYPE_CHECKING,
|
||||||
)
|
)
|
||||||
|
|
||||||
from async_generator import aclosing
|
from async_generator import aclosing
|
||||||
|
|
@ -74,6 +75,9 @@ from ._util import (
|
||||||
)
|
)
|
||||||
from .symbols import get_mkt_info
|
from .symbols import get_mkt_info
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from trio._core._run import Task
|
||||||
|
|
||||||
log = get_logger(
|
log = get_logger(
|
||||||
name=__name__,
|
name=__name__,
|
||||||
)
|
)
|
||||||
|
|
@ -349,7 +353,6 @@ async def wait_on_data_reset(
|
||||||
tuple[
|
tuple[
|
||||||
trio.CancelScope,
|
trio.CancelScope,
|
||||||
trio.Event,
|
trio.Event,
|
||||||
list[bool],
|
|
||||||
]
|
]
|
||||||
] = trio.TASK_STATUS_IGNORED,
|
] = trio.TASK_STATUS_IGNORED,
|
||||||
) -> bool:
|
) -> bool:
|
||||||
|
|
@ -393,9 +396,8 @@ async def wait_on_data_reset(
|
||||||
client: Client = proxy._aio_ns
|
client: Client = proxy._aio_ns
|
||||||
|
|
||||||
done = trio.Event()
|
done = trio.Event()
|
||||||
result: list[bool] = []
|
|
||||||
with trio.move_on_after(timeout) as cs:
|
with trio.move_on_after(timeout) as cs:
|
||||||
task_status.started((cs, done, result))
|
task_status.started((cs, done))
|
||||||
|
|
||||||
log.warning(
|
log.warning(
|
||||||
'Sending DATA RESET request:\n'
|
'Sending DATA RESET request:\n'
|
||||||
|
|
@ -411,7 +413,6 @@ async def wait_on_data_reset(
|
||||||
'NO VNC DETECTED!\n'
|
'NO VNC DETECTED!\n'
|
||||||
'Manually press ctrl-alt-f on your IB java app'
|
'Manually press ctrl-alt-f on your IB java app'
|
||||||
)
|
)
|
||||||
result.append(False)
|
|
||||||
done.set()
|
done.set()
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
@ -426,7 +427,6 @@ async def wait_on_data_reset(
|
||||||
]:
|
]:
|
||||||
await ev.wait()
|
await ev.wait()
|
||||||
log.info(f"{name} DATA RESET")
|
log.info(f"{name} DATA RESET")
|
||||||
result.append(True)
|
|
||||||
done.set()
|
done.set()
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
@ -435,12 +435,12 @@ async def wait_on_data_reset(
|
||||||
'Data reset task canceled?'
|
'Data reset task canceled?'
|
||||||
)
|
)
|
||||||
|
|
||||||
result.append(False)
|
|
||||||
done.set()
|
done.set()
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
_data_reset_lock = trio.Lock()
|
_data_resetter_task: Task | None = None
|
||||||
|
_failed_resets: int = 0
|
||||||
|
|
||||||
|
|
||||||
async def get_bars(
|
async def get_bars(
|
||||||
|
|
@ -479,11 +479,8 @@ async def get_bars(
|
||||||
`.ib.api.Client` methods.
|
`.ib.api.Client` methods.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
global _data_resetter_task, _failed_resets
|
||||||
nodatas_count: int = 0
|
nodatas_count: int = 0
|
||||||
cancelled_count: int = 0
|
|
||||||
failed_resets: int = 0
|
|
||||||
pacing_reset_pending: bool = False
|
|
||||||
pacing_reset_completed: bool = False
|
|
||||||
|
|
||||||
data_cs: trio.CancelScope | None = None
|
data_cs: trio.CancelScope | None = None
|
||||||
result: tuple[
|
result: tuple[
|
||||||
|
|
@ -496,16 +493,10 @@ async def get_bars(
|
||||||
|
|
||||||
async def query():
|
async def query():
|
||||||
|
|
||||||
nonlocal result, data_cs, end_dt
|
global _failed_resets
|
||||||
nonlocal nodatas_count, cancelled_count, failed_resets
|
nonlocal result, data_cs, end_dt, nodatas_count
|
||||||
nonlocal pacing_reset_pending, pacing_reset_completed
|
|
||||||
|
|
||||||
while True:
|
while _failed_resets < max_failed_resets:
|
||||||
if failed_resets >= max_failed_resets:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'IB history reset failed {failed_resets} '
|
|
||||||
f'times for {fqme}'
|
|
||||||
)
|
|
||||||
try:
|
try:
|
||||||
(
|
(
|
||||||
bars,
|
bars,
|
||||||
|
|
@ -537,10 +528,6 @@ async def get_bars(
|
||||||
)
|
)
|
||||||
# NOTE: REQUIRED to pass back value..
|
# NOTE: REQUIRED to pass back value..
|
||||||
result = None
|
result = None
|
||||||
failed_resets = 0
|
|
||||||
result_ready.set()
|
|
||||||
if data_cs:
|
|
||||||
data_cs.cancel()
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
# not enough bars signal, likely due to venue
|
# not enough bars signal, likely due to venue
|
||||||
|
|
@ -600,7 +587,6 @@ async def get_bars(
|
||||||
first_dt,
|
first_dt,
|
||||||
last_dt,
|
last_dt,
|
||||||
)
|
)
|
||||||
failed_resets = 0
|
|
||||||
|
|
||||||
# signal data reset loop parent task
|
# signal data reset loop parent task
|
||||||
result_ready.set()
|
result_ready.set()
|
||||||
|
|
@ -656,12 +642,6 @@ async def get_bars(
|
||||||
'Query cancelled by IB (:eyeroll:):\n'
|
'Query cancelled by IB (:eyeroll:):\n'
|
||||||
f'{err.message}'
|
f'{err.message}'
|
||||||
)
|
)
|
||||||
cancelled_count += 1
|
|
||||||
if cancelled_count >= max_failed_resets:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'IB cancelled {cancelled_count} history '
|
|
||||||
f'queries for {fqme}'
|
|
||||||
)
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
elif (
|
elif (
|
||||||
|
|
@ -682,7 +662,6 @@ async def get_bars(
|
||||||
)
|
)
|
||||||
|
|
||||||
client = proxy._aio_ns.ib.client
|
client = proxy._aio_ns.ib.client
|
||||||
pacing_reset_pending = True
|
|
||||||
|
|
||||||
# cancel any existing reset task
|
# cancel any existing reset task
|
||||||
if data_cs:
|
if data_cs:
|
||||||
|
|
@ -690,81 +669,69 @@ async def get_bars(
|
||||||
data_cs.cancel()
|
data_cs.cancel()
|
||||||
|
|
||||||
# spawn new data reset task
|
# spawn new data reset task
|
||||||
async with _data_reset_lock:
|
data_cs, reset_done = await tn.start(
|
||||||
(
|
|
||||||
data_cs,
|
|
||||||
reset_done,
|
|
||||||
reset_result,
|
|
||||||
) = await tn.start(
|
|
||||||
partial(
|
partial(
|
||||||
wait_on_data_reset,
|
wait_on_data_reset,
|
||||||
proxy,
|
proxy,
|
||||||
reset_type='connection'
|
reset_type='connection'
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
await reset_done.wait()
|
if reset_done:
|
||||||
if reset_result[0]:
|
_failed_resets = 0
|
||||||
failed_resets = 0
|
|
||||||
else:
|
else:
|
||||||
failed_resets += 1
|
_failed_resets += 1
|
||||||
pacing_reset_pending = False
|
|
||||||
pacing_reset_completed = True
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
else:
|
else:
|
||||||
raise
|
raise
|
||||||
|
|
||||||
|
# TODO: make this global across all history task/requests
|
||||||
|
# such that simultaneous symbol queries don't try data resettingn
|
||||||
|
# too fast..
|
||||||
|
unset_resetter: bool = False
|
||||||
async with (
|
async with (
|
||||||
tractor.trionics.collapse_eg(),
|
tractor.trionics.collapse_eg(),
|
||||||
trio.open_nursery() as tn
|
trio.open_nursery() as tn
|
||||||
):
|
):
|
||||||
|
|
||||||
# Run history until a result or bounded reset failure.
|
# start history request that we allow
|
||||||
|
# to run indefinitely until a result is acquired
|
||||||
tn.start_soon(query)
|
tn.start_soon(query)
|
||||||
|
|
||||||
# Serialize farm resets across every history requester.
|
# start history reset loop which waits up to the timeout
|
||||||
|
# for a result before triggering a data feed reset.
|
||||||
while not result_ready.is_set():
|
while not result_ready.is_set():
|
||||||
|
|
||||||
with trio.move_on_after(feed_reset_timeout):
|
with trio.move_on_after(feed_reset_timeout):
|
||||||
await result_ready.wait()
|
await result_ready.wait()
|
||||||
break
|
break
|
||||||
|
|
||||||
async with _data_reset_lock:
|
if _data_resetter_task:
|
||||||
if result_ready.is_set():
|
# don't double invoke the reset hack if another
|
||||||
break
|
# requester task already has it covered.
|
||||||
if pacing_reset_completed:
|
|
||||||
pacing_reset_completed = False
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
(
|
else:
|
||||||
data_cs,
|
_data_resetter_task = trio.lowlevel.current_task()
|
||||||
reset_done,
|
unset_resetter: bool = True
|
||||||
reset_result,
|
|
||||||
) = await tn.start(
|
# spawn new data reset task
|
||||||
|
data_cs, reset_done = await tn.start(
|
||||||
partial(
|
partial(
|
||||||
wait_on_data_reset,
|
wait_on_data_reset,
|
||||||
proxy,
|
proxy,
|
||||||
reset_type='data',
|
reset_type='data',
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
# sync wait on reset to complete
|
||||||
await reset_done.wait()
|
await reset_done.wait()
|
||||||
|
|
||||||
if (
|
_data_resetter_task = (
|
||||||
result_ready.is_set()
|
None
|
||||||
or
|
if unset_resetter
|
||||||
reset_result[0]
|
else _data_resetter_task
|
||||||
):
|
|
||||||
failed_resets = 0
|
|
||||||
elif pacing_reset_pending:
|
|
||||||
# Pacing recovery deliberately cancelled this reset
|
|
||||||
# and will consume the retry itself.
|
|
||||||
pass
|
|
||||||
else:
|
|
||||||
failed_resets += 1
|
|
||||||
if failed_resets >= max_failed_resets:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'IB history reset failed '
|
|
||||||
f'{failed_resets} times for {fqme}'
|
|
||||||
)
|
)
|
||||||
|
assert result
|
||||||
return (
|
return (
|
||||||
result,
|
result,
|
||||||
data_cs is not None,
|
data_cs is not None,
|
||||||
|
|
|
||||||
|
|
@ -501,7 +501,7 @@ async def update_ledger_from_api_trades(
|
||||||
for fill in fills:
|
for fill in fills:
|
||||||
con: Contract = fill.contract
|
con: Contract = fill.contract
|
||||||
conid: str = con.conId
|
conid: str = con.conId
|
||||||
pexch: str|None = con.primaryExchange or con.exchange
|
pexch: str | None = con.primaryExchange
|
||||||
|
|
||||||
if not pexch:
|
if not pexch:
|
||||||
cons = await client.get_con(conid=conid)
|
cons = await client.get_con(conid=conid)
|
||||||
|
|
|
||||||
|
|
@ -639,40 +639,12 @@ async def get_mkt_info(
|
||||||
|
|
||||||
# if possible register the bs_mktid to the just-built
|
# if possible register the bs_mktid to the just-built
|
||||||
# mkt so that it can be retreived by order mode tasks later.
|
# mkt so that it can be retreived by order mode tasks later.
|
||||||
|
# TODO NOTE: this is going to be problematic if/when we split
|
||||||
|
# out the datatd vs. brokerd actors since the mktmap lookup
|
||||||
|
# table will now be inaccessible..
|
||||||
if proxy is not None:
|
if proxy is not None:
|
||||||
cache_contract(
|
client: Client = proxy._aio_ns
|
||||||
proxy._aio_ns,
|
client._contracts[mkt.bs_fqme] = con
|
||||||
mkt,
|
client._cons2mkts[con] = mkt
|
||||||
con,
|
|
||||||
)
|
|
||||||
|
|
||||||
return mkt, details
|
return mkt, details
|
||||||
|
|
||||||
|
|
||||||
def cache_contract(
|
|
||||||
client: Client,
|
|
||||||
mkt: MktPair,
|
|
||||||
con: ibis.Contract,
|
|
||||||
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Register a (qualified) contract + mkt-info pair on the
|
|
||||||
given (api) `Client`'s actor-local lookup tables.
|
|
||||||
|
|
||||||
Cached under BOTH fqme key-forms since consumers vary:
|
|
||||||
- `mkt.bs_fqme` (eg. 'nvda.nasdaq'): read by
|
|
||||||
`Client.submit_limit()` for order requests,
|
|
||||||
- `mkt.fqme` (eg. 'nvda.nasdaq.ib'): read by the
|
|
||||||
fill-time pp-update (symcache-backup-table) path in
|
|
||||||
`.broker.emit_pp_update()`.
|
|
||||||
|
|
||||||
NOTE: post the (datad|brokerd)-actor-split this MUST be
|
|
||||||
run (in the trading actor) for every mkt either eagerly
|
|
||||||
at `.broker.open_trade_dialog()` startup or lazily per
|
|
||||||
order request; there is no in-proc feed setup doing it
|
|
||||||
implicitly anymore!
|
|
||||||
|
|
||||||
'''
|
|
||||||
client._contracts[mkt.bs_fqme] = con
|
|
||||||
client._contracts[mkt.fqme] = con
|
|
||||||
client._cons2mkts[con] = mkt
|
|
||||||
|
|
|
||||||
|
|
@ -33,21 +33,13 @@ from typing import (
|
||||||
)
|
)
|
||||||
|
|
||||||
import exchange_calendars as xcals
|
import exchange_calendars as xcals
|
||||||
from exchange_calendars.errors import (
|
|
||||||
InvalidCalendarName,
|
|
||||||
)
|
|
||||||
from pendulum import (
|
from pendulum import (
|
||||||
parse,
|
|
||||||
now,
|
now,
|
||||||
Duration,
|
Duration,
|
||||||
Interval,
|
Interval,
|
||||||
Time,
|
Time,
|
||||||
)
|
)
|
||||||
|
|
||||||
from piker.log import get_logger
|
|
||||||
|
|
||||||
log = get_logger(__name__)
|
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from ib_async import (
|
from ib_async import (
|
||||||
TradingSession,
|
TradingSession,
|
||||||
|
|
@ -64,22 +56,6 @@ if TYPE_CHECKING:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def is_expired(
|
|
||||||
con_deats: ContractDetails,
|
|
||||||
) -> bool:
|
|
||||||
'''
|
|
||||||
Simple predicate whether the provided contract-deats match and
|
|
||||||
already lifetime-terminated instrument.
|
|
||||||
|
|
||||||
'''
|
|
||||||
expiry_str: str = con_deats.realExpirationDate
|
|
||||||
if not expiry_str:
|
|
||||||
return False
|
|
||||||
|
|
||||||
expiry_dt: datetime = parse(expiry_str)
|
|
||||||
return expiry_dt.date() >= now().date()
|
|
||||||
|
|
||||||
|
|
||||||
def has_weekend(
|
def has_weekend(
|
||||||
period: Interval,
|
period: Interval,
|
||||||
) -> bool:
|
) -> bool:
|
||||||
|
|
@ -114,28 +90,13 @@ def has_holiday(
|
||||||
con.exchange
|
con.exchange
|
||||||
)
|
)
|
||||||
|
|
||||||
# XXX, ad-hoc handle any IB exchange which are
|
# XXX, ad-hoc handle any IB exchange which are non-std
|
||||||
# non-std via lookup table..
|
# via lookup table..
|
||||||
std_exch: str = {
|
std_exch: dict = {
|
||||||
'ARCA': 'ARCX',
|
'ARCA': 'ARCX',
|
||||||
}.get(exch, exch)
|
}.get(exch, exch)
|
||||||
|
|
||||||
try:
|
cal: ExchangeCalendar = xcals.get_calendar(std_exch)
|
||||||
cal: ExchangeCalendar = xcals.get_calendar(
|
|
||||||
std_exch
|
|
||||||
)
|
|
||||||
except InvalidCalendarName:
|
|
||||||
# venue has no `exchange_calendars` entry
|
|
||||||
# (eg. IDEALPRO for forex, PAXOS for
|
|
||||||
# crypto) -> not a holiday by default since
|
|
||||||
# weekends are already handled by
|
|
||||||
# `has_weekend()`.
|
|
||||||
log.warning(
|
|
||||||
f'No exchange cal for {std_exch!r},'
|
|
||||||
f' skipping holiday check..\n'
|
|
||||||
)
|
|
||||||
return False
|
|
||||||
|
|
||||||
end: datetime = period.end
|
end: datetime = period.end
|
||||||
# _start: datetime = period.start
|
# _start: datetime = period.start
|
||||||
# ?TODO, can rm ya?
|
# ?TODO, can rm ya?
|
||||||
|
|
@ -209,22 +170,7 @@ def sesh_times(
|
||||||
get the (day-agnostic) times for the start/end.
|
get the (day-agnostic) times for the start/end.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
# ?TODO, lookup the next front contract instead?
|
earliest_sesh: Interval = next(iter_sessions(con_deats))
|
||||||
if is_expired(con_deats):
|
|
||||||
raise ValueError(
|
|
||||||
f'Contract is already expired!\n'
|
|
||||||
f'Choose an active alt contract instead.\n'
|
|
||||||
f'con_deats: {con_deats!r}\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
maybe_sessions: list[Interval] = list(iter_sessions(con_deats))
|
|
||||||
if not maybe_sessions:
|
|
||||||
raise ValueError(
|
|
||||||
f'Contract has no trading-session info?\n'
|
|
||||||
f'con_deats: {con_deats!r}\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
earliest_sesh: Interval = maybe_sessions[0]
|
|
||||||
return (
|
return (
|
||||||
earliest_sesh.start.time(),
|
earliest_sesh.start.time(),
|
||||||
earliest_sesh.end.time(),
|
earliest_sesh.end.time(),
|
||||||
|
|
@ -265,13 +211,7 @@ def is_venue_closure(
|
||||||
'''
|
'''
|
||||||
open: Time
|
open: Time
|
||||||
close: Time
|
close: Time
|
||||||
maybe_oc: tuple|None = sesh_times(con_deats)
|
open, close = sesh_times(con_deats)
|
||||||
if maybe_oc is None:
|
|
||||||
# XXX, should never get here.
|
|
||||||
breakpoint()
|
|
||||||
return False
|
|
||||||
|
|
||||||
open, close = maybe_oc
|
|
||||||
|
|
||||||
# ensure times are in mkt-native timezone
|
# ensure times are in mkt-native timezone
|
||||||
tz: str = con_deats.timeZoneId
|
tz: str = con_deats.timeZoneId
|
||||||
|
|
|
||||||
|
|
@ -66,24 +66,10 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# per-daemon-kind (sub)mod groups: declares which of our
|
# tractor RPC enable arg
|
||||||
# submods host the eps run by each daemon-actor kind as
|
__enable_modules__: list[str] = [
|
||||||
# defined by `piker.data.validate._eps`.
|
|
||||||
_brokerd_mods: list[str] = [
|
|
||||||
'api',
|
'api',
|
||||||
'broker',
|
'broker',
|
||||||
]
|
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
|
||||||
'api',
|
|
||||||
'feed',
|
'feed',
|
||||||
'symbols',
|
'symbols',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
|
||||||
_brokerd_mods
|
|
||||||
+
|
|
||||||
_datad_mods
|
|
||||||
))
|
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,6 @@ import hashlib
|
||||||
import hmac
|
import hmac
|
||||||
import base64
|
import base64
|
||||||
import tractor
|
import tractor
|
||||||
# from tractor._exceptions import reg_err_types
|
|
||||||
import trio
|
import trio
|
||||||
|
|
||||||
from piker import config
|
from piker import config
|
||||||
|
|
@ -53,7 +52,6 @@ from piker.brokers._util import (
|
||||||
SymbolNotFound,
|
SymbolNotFound,
|
||||||
BrokerError,
|
BrokerError,
|
||||||
DataThrottle,
|
DataThrottle,
|
||||||
get_or_raise_on_pair_schema_mismatch,
|
|
||||||
)
|
)
|
||||||
from piker.accounting import Transaction
|
from piker.accounting import Transaction
|
||||||
from piker.log import get_logger
|
from piker.log import get_logger
|
||||||
|
|
@ -109,37 +107,15 @@ def get_kraken_signature(
|
||||||
return sigdigest.decode()
|
return sigdigest.decode()
|
||||||
|
|
||||||
|
|
||||||
# class InvalidKey(ValueError):
|
class InvalidKey(ValueError):
|
||||||
# '''
|
'''
|
||||||
# EAPI:Invalid key
|
EAPI:Invalid key
|
||||||
|
This error is returned when the API key used for the call is
|
||||||
|
either expired or disabled, please review the API key in your
|
||||||
|
Settings -> API tab of account management or generate a new one
|
||||||
|
and update your application.
|
||||||
|
|
||||||
# This error is returned when the API key used for the call is
|
'''
|
||||||
# either expired or disabled, please review the API key in your
|
|
||||||
# Settings -> API tab of account management or generate a new one
|
|
||||||
# and update your application.
|
|
||||||
|
|
||||||
# '''
|
|
||||||
|
|
||||||
|
|
||||||
# class InvalidSession(RuntimeError):
|
|
||||||
# '''
|
|
||||||
# ESession:Invalid session
|
|
||||||
|
|
||||||
# This error is returned when the ws API key used for an authenticated
|
|
||||||
# sub/endpoint becomes stale, normally after a sufficient network
|
|
||||||
# disconnect/outage.
|
|
||||||
|
|
||||||
# Normally the sub will need to be restarted, likely re-init of the
|
|
||||||
# auth handshake sequence.
|
|
||||||
|
|
||||||
# '''
|
|
||||||
# subscription: dict
|
|
||||||
|
|
||||||
|
|
||||||
# reg_err_types([
|
|
||||||
# InvalidKey,
|
|
||||||
# InvalidSession,
|
|
||||||
# ])
|
|
||||||
|
|
||||||
|
|
||||||
class Client:
|
class Client:
|
||||||
|
|
@ -167,16 +143,18 @@ class Client:
|
||||||
config: dict[str, str],
|
config: dict[str, str],
|
||||||
httpx_client: httpx.AsyncClient,
|
httpx_client: httpx.AsyncClient,
|
||||||
|
|
||||||
key_descr: str = '',
|
name: str = '',
|
||||||
api_key: str = '',
|
api_key: str = '',
|
||||||
secret: str = ''
|
secret: str = ''
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
||||||
self._sesh: httpx.AsyncClient = httpx_client
|
self._sesh: httpx.AsyncClient = httpx_client
|
||||||
self._key_descr = key_descr
|
|
||||||
|
self._name = name
|
||||||
self._api_key = api_key
|
self._api_key = api_key
|
||||||
self._secret = secret
|
self._secret = secret
|
||||||
|
|
||||||
self.conf: dict[str, str] = config
|
self.conf: dict[str, str] = config
|
||||||
self._ws_token: str|None = None
|
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def pairs(self) -> dict[str, Pair]:
|
def pairs(self) -> dict[str, Pair]:
|
||||||
|
|
@ -261,39 +239,6 @@ class Client:
|
||||||
|
|
||||||
return balances
|
return balances
|
||||||
|
|
||||||
async def get_ws_token(
|
|
||||||
self,
|
|
||||||
params: dict = {},
|
|
||||||
force_renewal: bool = False,
|
|
||||||
) -> str:
|
|
||||||
'''
|
|
||||||
Get websocket token for authenticated data stream and cache
|
|
||||||
it for reuse.
|
|
||||||
|
|
||||||
Assert a value was actually received before return.
|
|
||||||
|
|
||||||
'''
|
|
||||||
if (
|
|
||||||
not force_renewal
|
|
||||||
and
|
|
||||||
self._ws_token
|
|
||||||
):
|
|
||||||
return self._ws_token
|
|
||||||
|
|
||||||
resp = await self.endpoint(
|
|
||||||
'GetWebSocketsToken',
|
|
||||||
{},
|
|
||||||
)
|
|
||||||
if err := resp.get('error'):
|
|
||||||
raise BrokerError(err)
|
|
||||||
|
|
||||||
# resp token for ws init
|
|
||||||
token: str = resp['result']['token']
|
|
||||||
assert token
|
|
||||||
|
|
||||||
self._ws_token: str = token
|
|
||||||
return token
|
|
||||||
|
|
||||||
async def get_assets(
|
async def get_assets(
|
||||||
self,
|
self,
|
||||||
reload: bool = False,
|
reload: bool = False,
|
||||||
|
|
@ -557,16 +502,7 @@ class Client:
|
||||||
|
|
||||||
# NOTE: always cache in pairs tables for faster lookup
|
# NOTE: always cache in pairs tables for faster lookup
|
||||||
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
||||||
# pair = Pair(xname=xkey, **data)
|
pair = Pair(xname=xkey, **data)
|
||||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
|
||||||
pair_type=Pair,
|
|
||||||
fields_data=dict(
|
|
||||||
xname=xkey,
|
|
||||||
**data,
|
|
||||||
),
|
|
||||||
provider_name='kraken',
|
|
||||||
# api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
|
||||||
)
|
|
||||||
|
|
||||||
# register the above `Pair` structs for all
|
# register the above `Pair` structs for all
|
||||||
# key-sets/monikers: a set of 4 (frickin) tables
|
# key-sets/monikers: a set of 4 (frickin) tables
|
||||||
|
|
@ -732,13 +668,7 @@ class Client:
|
||||||
|
|
||||||
@acm
|
@acm
|
||||||
async def get_client() -> Client:
|
async def get_client() -> Client:
|
||||||
'''
|
|
||||||
Load and deliver a `.kraken.api.Client`.
|
|
||||||
|
|
||||||
When defined, inject any config delivered from the user's
|
|
||||||
`brokers.toml` config file.
|
|
||||||
|
|
||||||
'''
|
|
||||||
conf: dict[str, Any] = get_config()
|
conf: dict[str, Any] = get_config()
|
||||||
async with httpx.AsyncClient(
|
async with httpx.AsyncClient(
|
||||||
base_url=_url,
|
base_url=_url,
|
||||||
|
|
@ -749,14 +679,13 @@ async def get_client() -> Client:
|
||||||
# connections=4
|
# connections=4
|
||||||
) as trio_client:
|
) as trio_client:
|
||||||
if conf:
|
if conf:
|
||||||
api_key_descr: str = conf['key_descr']
|
|
||||||
client = Client(
|
client = Client(
|
||||||
conf,
|
conf,
|
||||||
httpx_client=trio_client,
|
httpx_client=trio_client,
|
||||||
|
|
||||||
# TODO: don't break these up and just do internal
|
# TODO: don't break these up and just do internal
|
||||||
# conf lookups instead..
|
# conf lookups instead..
|
||||||
key_descr=api_key_descr,
|
name=conf['key_descr'],
|
||||||
api_key=conf['api_key'],
|
api_key=conf['api_key'],
|
||||||
secret=conf['secret']
|
secret=conf['secret']
|
||||||
)
|
)
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue