Add a new `snippets/claude_debug_helper.py` to
provide a programmatic interface to `tractor.pause()` debugger
sessions for incremental data inspection matching the interactive UX
but able to be run by `claude` "offline" since it can't seem to feed
stdin (so it claims) to the `pdb` instance due to lack of ability to
allocate a tty internally.
The script-wrapper is based on `tractor`'s `tests/devx/` suite's use of
`pexpect` patterns for driving `pdbp` prompts and thus enables
automated-offline execution of REPL-inspection commands **without**
using incremental-realtime output capture (like a human would use it).
Features:
- `run_pdb_commands()`: batch command execution
- `InteractivePdbSession`: context manager for step-by-step REPL interaction
- `expect()` wrapper: timeout handling with buffer display
- Proper stdin/stdout handling via `pexpect.spawn()`
Example usage:
```python
from debug_helper import InteractivePdbSession
with InteractivePdbSession(
cmd='piker store ldshm zecusdt.usdtm.perp.binance'
) as session:
session.run('deduped.shape')
session.run('step_gaps.shape')
```
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Polars tightened type safety for `.dt` accessor methods requiring
`total_*` methods for duration types vs datetime component accessors
like `day()` which now only work on datetime dtypes.
`detect_time_gaps()` in `.tsp._anal` was calling `.dt.day()`
on `dt_diff` column (a duration from `.diff()`) which throws
`InvalidOperationError` on modern polars.
Changes:
- use f-string to add pluralization to map time unit strings to
`total_<unit>s` form for the new duration API.
- Handle singular/plural forms: 'day' -> 'days' -> 'total_days'
- Ensure trailing 's' before applying 'total_' prefix
Also updates inline comments explaining the polars type distinction
between datetime components vs duration totals.
Fixes `piker store ldshm` crashes on datasets with time gaps.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
The `await feed_is_live.wait()` is more or less pointless and would only
cause slower startup afaig (as-far-as-i-grok) so i'm masking it here.
This also removes the final `strict_exception_groups=False` use from the
non-tests code base, flipping to the `tractor.trionics` collapser once
and for all!
Refmt the "connection-dropped" error-log in `Sampler`'s broadcast loop
to show error type first, then the IPC context details; mks it all
easier to grok/less-noisy on console imo.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Add outer `try/except` around the nursery block in
`open_autorecon_ws()` to catch any `NoBsWs.recon_errors` that
escape the inner reconnect loop, logging a warning instead of
propagating.
Also,
- correct `NoBsWs.recon_errors` typing to `tuple[Type[Exception]]`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Add timeout param to `.reset()` and `.send_msg()` to prevent
indefinite blocking on reconnect attempts. Shield reconnect
sleeps from cancellation to ensure we avoid any "finally footgun" type
scenarios where `trio.Cancelled` masks an underlying exc per,
- https://github.com/goodboy/tractor/pull/387
- https://github.com/goodboy/tractor/pull/391
Deats,
- add `timeout` param to `.reset()`, return `bool` for success
- add `timeout=3` default to `.send_msg()` for reconnect wait
- shield `.reset()` call in `.send_msg()` error handler
- log warning when reconnect timeout exceeded
- shield throttled sleeps in `_reconnect_forever()` error paths
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
In both the ems and sampler since on new `tractor` this is the
"wrapping" exception raised when the transport layer terminates early
but in a psuedo-"graceful" way, expected when a peer actors disconnect.
Previously we were crashing in this case since old `tractor` just raised
the underlying `trio`-source-exceptions verbatim.
Also,
- use `Aid.reprol()` in log msgs vs old `.chan.uid` refs
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Extract root-most widget to resolve (various) `.ui` import cycles
when the type is declared on `Struct`s..
Deats,
- flip to `from ._widget import GodWidget`.
- move `Feed` + `Flume` imports to TYPE_CHECKING in `._chart`
- drop unused `trio` import from `._chart`
- fix docstring typo: "datums```" -> "`datums``"
- change `print()` to `log.warning()` for global step msg
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Drop the local (and original) `Struct` impl from `piker.types` in favour
of `tractor`'s version now that it's been upstreamed.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Expand read-race warning log for clarity, add TODO for reading
`tractor` transport config from `conf.toml`, and reflow docstring
in `open_vlm_displays()`.
Also,
- whitespace cleanup: `Type | None` -> `Type|None`
- clarify "Volume" -> "Vlm (volume)" in docstr
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Expose `get_godw()` helper to retrieve the central `GodWidget`
instance from anywhere in the UI code. Set the singleton in
`_async_main()` on startup.
Also,
- add docstring to `run_qtractor()` explaining trio guest mode
- type annotate `instance: GodWidget` in `run_qtractor()`
- import reorg in `._app` for cleaner grouping
- whitespace cleanup: `Type | None` -> `Type|None` throughout
- fix bitwise-or alignment: `Flag | Other` -> `Flag|Other`
(this commit-msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Since i'm seeing IB records with a `None` value and i don't want to be
debugging every time order-mode boots up..
Also use `pdb=debug` in `.open_ledger_dfs()`
Note, this had conflicts on `piker/accounting/calc.py` when rebasing
onto the refactored `brokers_refinery` history which were resolved
manually!
To start this is just a shell for the test, there's no checking logic
yet.. put it as `test_accounting.test_ib_account_with_duplicated_mktids()`.
The test is composed for now to be completely runtime-free using only
the offline txn-ledger / symcache / account loading APIs, ideally we
fill in the activated symbology-data-runtime cases once we figure a sane
way to handle incremental symcache updates for backends like IB..
To actually fill the test out with real checks we still need to,
- extract the problem account file from my ib.algopape into the test
harness data.
- pick some contracts with multiple fqmes despite a single bs_mktid and
ensure they're aggregated as a single `Position` as well as,
* ideally de-duplicating txns from the account file section for the
mkt..
* warning appropriately about greater-then-one fqme for the bs_mktid
and providing a way for the ledger re-writing to choose the
appropriate `<venue>` as the "primary" when the
data-symbology-runtime is up and possibly use it to incrementally
update the IB symcache and store offline for next use?
Since under `trio`-cancellation the `.remove()` is a checkpoint and will
be masked by a taskc AND we **always want to remove the rect** despite
the surrounding teardown conditions.
In the `translate_and_relay_brokerd_events()` loop task that is, such
that we never crash on a `status_msg = book._active.pop(oid)` in the
'closed' status handler whenever a double removal happens.
Turns out there were unforeseen races here when a benign backend error
would cause an order-mode dialog to be cancelled (incorrectly) and then
a UI side `.on_cancel()` would trigger too-early removal from the
`book._active` table despite the backend sending an actual 'closed'
event (much) later, this would crash on the now missing entry..
So instead we now,
- obviously use `book._active.pop(oid, None)`
- emit a `log.warning()` (not info lol) on a null-read and with a less
"one-line-y" message explaining the double removal and maybe *why*.
For backends which opt to set the new `BrokerdPosition.bs_mktid` field,
give (matching logic) priority to it such that even if the `.symbol`
field doesn't match the mkt currently focussed on chart, it will
always match on a provider's own internal asset-mapping-id. The original
fallback logic for `.fqme` matching is left as is.
As an example with IB, a qqq.nasdaq.ib txn may have been filled on
a non-primary venue as qqq.directedea.ib, in this case if the mkt is
displayed and focused on chart we want the **entire position info** to
be overlayed by the `OrderMode` UX without discrepancy.
Other refinements,
- improve logging and add a detailed edge-case-comment around the
`.on_fill()` handler to clarify where if a benign 'error' msg is
relayed from a backend it will cause the UI to operate as though the
order **was not-cleared/cancelled** since the `.on_cancel()` handler
will have likely been called just before, popping the `.dialogs`
entry. Return `bool` to indicate whether the UI removed-lines
/ added-fill-arrows.
- inverse the `return` branching logic in `.on_cancel()` to reduce
indent.
- add a very loud `log.error()` in `Status(resp='error')` case-block
ensuring the console yells about the order being cancelled, also
a todo for the weird msg-field recursion nonsense..
Such that backends can deliver their own internal unique
`MktPair.bs_mktid` when they can't seem to get it right via the
`.fqme: str` export.. (COUGH ib, you piece of sh#$).
Also add todo for possibly replacing the msg with a `Position.summary()`
"snapshot" as a better and more rigorously generated wire-ready msg.
Despite a `.bs_mktid` ideally being a bijection with `MktPair.fqme`
values, apparently some backends (cough IB) will switch the .<venue>`
part in txn records resulting in multiple account-conf-file sections for
the same dst asset. Obviously that means we can't allocate new
`Position` entries keyed by that `bs_mktid`, instead be sure to **update
them instead**!
Deats,
- add case logic to avoid pp overwrites using a `pp_objs.get()` check.
- warn on duplicated pos entries whenever the current account-file
entry's `mkt` doesn't match the pre-existing position's.
- mk `Position.add_clear()` return a `bool` indicating if the record was
newly added, warn when it was already existing/added prior.
Also,
- drop the already deprecated `open_pps()`, also from sub-pkg exports.
- draft TODO for `Position.summary()` idea as a replacement for
`BrokerdPosition`-msgs.
That is inside embedded `.accounting.calc.dyn_parse_to_dt()` closure add
an optional `_invalid: list` param to where we can report
bad-timestamped records which we instead override and return as
`from_timestamp(0.)` (when the parser loop falls through) and report
later (in summary ) from the `.accounting.calc.iter_by_dt()` caller. Add
some logging and an optional debug block for future tracing.
NOTE, this commit was re-edited during a conflict between the orig
branches: `dev/binance_api_3.1` & `dev/alt_tpts_for_perf`.
For `[ib]` adjust content to match changes to the
`dockering/ib/README.rst` and for `[deribit]` toss in the WIP options
related params for anyone who wants to play around with @nt's work.
Such that the `brokers.toml` can contain any of the following
<port> = dict|tuple styles,
```toml
[ib.vnc_addrs]
4002 = {host = 'localhost', port = 5900, pw = 'doggy'} # host, port, pw
4002 = {host = 'localhost', port = 5900} # host, port, pw
4002 = ['localhost', 5900] # host, port, pw
```
With the first line demonstrating a vnc-server password (as normally set
via a `.env` file in the `dockering/ib/` subdir) with the `pw =` field.
This obviously removes the hardcoded `'doggy'` password from prior.
Impl details in `.brokers.ib._util`:
- pass the `ib.api.Client` down into `vnc_click_hack()` doing all config
reading within and removing host, port unpacking in the callingn
`data_reset_hack()`.
- also pass the client `try_xdo_manual()` and comment (with plans to
remove) the recently added localhost-only fallback section since
we now have a fully working py vnc client again with `pyvnc` B)
- in `vnc_click_hack()` match for all the possible config line styles
and,
* pass any `pw` field to `pyvncVNCConfig`,
* continue matching host, port without password,
* fallthrough to raising a val-err when neither ^ match.
For the new github image, a high-level look at its basic
features/usage/docs and prosing around our expected default usage with
the `piker.brokers.ib` backend.
It actually works for vncAuth(2) (thank god!) which the previous
`asyncvnc` **did not**, and seems to be mostly based on the work
from the `asyncvnc` author anyway (so all my past efforts don't seem to
have been in vain XD).
NOTE, the below deats ended up being factored in earlier into the
`pyproject.toml` alongside nix(os) support needed for testing and
landing this history. As the such, the comments are the originals but
the changes are not.
Deats,
- switch to `pyvnc` async API (using `asyncio` again obvi) in
`.ib._util._vnc_click_hack()`.
- add `pyvnc` as src installed dep from GH.
- drop `asyncvnc` as dep.
Other,
- update `pytest` version range to avoid weird auto-load plugin exposed
by `xonsh`?
- add a `tool.pytest.ini_options` to project file with vars to,
- disable that^ `xonsh` plug using `addopts = '-p no:xonsh'`.
- set a `testpaths` to avoid running anything but that subdir.
- try out the `'progress'` style console output (does it work?).
Gud Boi
Tyler Goodlet