The topology flip: all data-feed consumers now route to the
new `datad.<broker>` sibling daemon; `brokerd` becomes
trading-only and is ONLY ever booted lazily by `emsd`'s
`open_brokerd_dialog()` (see prior commit). Chart-only and
paper sessions run with zero (live, credentialed) `brokerd`
procs B)
Deats,
- `open_feed()` -> `maybe_spawn_datad()` (NB: imported
relative-direct from `._daemon` to dodge a partial-init
cycle via `piker.service`); flip the `open_feed_bus()`
actor-name assert to `'datad'`; comment sweep.
- slim `_setup_persistent_brokerd()` to a trading-only
fixture: console logging + pinned-open ctx; the feed-bus
alloc moves to `_setup_persistent_datad()` and backend
`open_trade_dialog()` ctxs own their own task trees.
(the `piker ledger` ad-hoc actor enters this same slimmed
fixture - exactly what it needs.)
- repoint data-flavoured spawn sites to `maybe_spawn_datad`:
`.ui._app` symbol-search (+ rename
`install_brokerd_search` -> `install_datad_search`),
`.brokers.core.symbol_search()`, `.brokers.cli`
`brokercheck`/`record`, legacy kivy `.ui.cli` +
`option_chain`'s `wait_for_actor()`.
- invert `tests.test_services` expectations: feed/EMS-paper
flows must spawn `datad.kraken` and `paperboi.kraken`
with an explicit negative assert that NO `brokerd.kraken`
service task exists.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First half of the `brokerd` split: a new per-provider
data-feed-only daemon-actor `datad.<broker>` to (soon) host
all `validate._eps['datad']` eps (live quotes, history
loading, symbology search) leaving `brokerd` for live order
ctl only. Purely additive; nothing routes through it yet.
Deats,
- new `piker.data._daemon` mod mirroring the
`.brokers._daemon` conventions (and the `samplerd`
sub-daemon precedent):
- `_setup_persistent_datad()` lifetime fixture owning the
actor-global `_FeedsBus` alloc.
- `datad_init()` building `enable_modules` from the
backend's `_datad_mods` (falling back to
`__enable_modules__` for not-yet-split backends) and
copying `_spawn_kwargs` (critical for `ib`'s
`infect_asyncio`).
- `spawn_datad()`/`maybe_spawn_datad()` wrapping
`Services` + `maybe_spawn_daemon()`.
- add `piker.data._daemon` to `_root_modules` so `pikerd`
can run `spawn_datad()` requests.
- re-export the spawn eps from `piker.service`.
- add `test_datad_spawn` verifying actor boot + service
registration via `ensure_service('datad.kraken')`.
Note the `Services`-based impl style deliberately mirrors
`spawn_brokerd()` so the eventual `tractor.hilevel`
`ServiceMngr` port (see the `service_mng_to_tractor`
branch's d8c21d44 prep work) lands symmetrically on both.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Continue the `repair_tests`-branch mission (already merged
in this stack's ancestry, see f4c4f1e2 which ported
`conftest.py`) by fixing the remaining drift breakage vs.
`tractor` git `main`; without these NOTHING boots since the
`tractor.Address` port in 604e5fcf.
Deats,
- normalize reg addrs via `wrap_address()` in
`open_pikerd()` before `.unwrap()`-ing; entries may be
raw `tuple`s when passed in from (test) client code.
- port `check_for_service()` to `query_actor(regaddr=)`
(was `arbiter_sockaddr=`) incl. its 2-tuple yield and
the now-required `open_registry(addrs=)` arg.
- `wait_for_actor(registry_addr=)` + `.chan.raddr.unwrap()`
raw-tuple compares in `test_runtime_boot` and
`ensure_service()`.
- update `run_test_w_cancel_method()` for modern `tractor`
cancel semantics: self-requested sub-service cancels are
absorbed (no `ContextCancelled` raised to the opener) and
single-exc groups collapse to a bare KBI.
- `RemoteActorError.boxed_type` (was `.type`) and
`Position.cumsize` (was `.size`) renames in tests.
- bump the paper-EMS startup budget 9 -> 19s; it includes
a live (kraken) symbology fetch so needs net headroom.
- woops, add the missing comma in `.deribit.api`'s
`tractor.trionics` import tuple..
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Switch all `test_feeds.py` parametrized fqmes from
legacy `<pair>.<broker>` to the current
`<pair>.<venue>.<broker>` schema (e.g.
`btcusdt.spot.binance`).
Deats,
- update binance fqmes: `btcusdt.binance` ->
`btcusdt.spot.binance`, same for `ethusdt`.
- update kraken fqmes: `ethusdt.kraken` ->
`ethusdt.spot.kraken`, `xbtusd.kraken` ->
`xbtusd.spot.kraken`.
- update cross-broker set similarly.
- comment out old fqmes with `!TODO` to later
validate raising on bad/legacy formats.
Also,
- reformat `if ci_env and not run_in_ci` to
multiline style.
- reformat `pytest.skip()` msg to multiline.
- add `?TODO` for symbology helper fn.
- drop stray `await tractor.pause()` in
`conftest.py`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Port test fixtures to match `tractor`'s updated
registry and channel APIs.
Deats,
- use `registry_addrs=[reg_addr]` (list) instead of
`registry_addr=reg_addr` in `maybe_open_pikerd()`.
- `wait_for_actor()` now takes `registry_addr=`
kwarg instead of `arbiter_sockaddr=`.
- access `portal.chan` (not `.channel`) and unwrap
remote addr via `raddr.unwrap()`.
- yield `raddr._host`/`raddr._port` instead of
tuple-indexing.
- drop random port generation; accept `reg_addr`
fixture from `tractor`'s builtin pytest plugin.
Also,
- add `reg_addr: tuple` param to `open_test_pikerd()`
fixture (sourced from `tractor._testing.pytest`).
- type-narrow `reg_addr` to `tuple[str, int|str]`.
- drop unused `import random`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
To start this is just a shell for the test, there's no checking logic
yet.. put it as `test_accounting.test_ib_account_with_duplicated_mktids()`.
The test is composed for now to be completely runtime-free using only
the offline txn-ledger / symcache / account loading APIs, ideally we
fill in the activated symbology-data-runtime cases once we figure a sane
way to handle incremental symcache updates for backends like IB..
To actually fill the test out with real checks we still need to,
- extract the problem account file from my ib.algopape into the test
harness data.
- pick some contracts with multiple fqmes despite a single bs_mktid and
ensure they're aggregated as a single `Position` as well as,
* ideally de-duplicating txns from the account file section for the
mkt..
* warning appropriately about greater-then-one fqme for the bs_mktid
and providing a way for the ledger re-writing to choose the
appropriate `<venue>` as the "primary" when the
data-symbology-runtime is up and possibly use it to incrementally
update the IB symcache and store offline for next use?
Topically, throughout various (seemingly) console-UX-affecting or benign
spots in the code base; nothing that required more intervention beyond
things superficial. A few spots also include `trio.Nursery` ref renames
(always to something with a `tn` in it) and log-level reductions to
quiet (benign) console noise oriented around issues meant to be solved
long..
Note there's still a couple spots i left with the loose-ify flag because
i haven't fully tested them without using the latest version of
`tractor.trionics.collapse_eg()`, but more then likely they should flip
over fine.
Avoids the really sloppy flag passing to `open_pikerd()` and allows for
separation of the individual docker daemon starts.
Also add a new `root_conf() -> Path` fixture which will open and load
the `dict` for the new root `conf.toml` file.
Since `.config.load()` was changed to not touch conf files by default
(without explicitly setting `touch_if_dne: bool`), this ensures both the
global module value is set and the `brokers.toml` file exists before
every test.
More or less a complete rework which allows passing a detailed
clearing/fills input and allows for *not* rebooting the runtime / ems
between each position check.
Some further enhancements:
- use (unit) fractional sizes to simulate both the more realistic and
more "complex position calculation" case; since this is crypto.
- add a no-fqme-found test.
- factor cross-session/offline pos storage (pps.toml) checks into
a `load_and_check_pos()` helper which does all entry loading directly
from a provided `BrokerdPosition` msg.
- use the new `OrderClient.send()` async api.
More or less we need to be able to audit not only simple "make trades
check pps.toml files" tests (which btw were great to get started!).
We also need more sophisticated and granular order mgmt and service
config scenarios,
- full e2e EMS msg flow verification
- multi-client (dis)connection scenarios and/or monitoring
- dark order clearing and offline storage
- accounting schema and position calcs detailing
As such, this is the beginning to "modularlizingz" the components needed
in the test harness to this end by breaking up the `OrderClient` control
flows vs. position checking logic so as to allow for more flexible test
scenario cases and likely `pytest` parametrizations over different
transaction sequences.
Not sure how this worked before but we need to also override the
`piker._config_dir: Path` in the root actor when running in `pytest`; my
guess is something in the old test suite was masking this problem after
the change to passing the dir path down through the runtime vars via
`tractor`?
Also this drops the ems related fixtures/factories since they're
specific enough to define in the clearing engine tests directly.
Gud Boi
di1ara
Tyler Goodlet