pikers
/
piker
4
1
Fork 0
Code Issues 12 Pull Requests 9 Packages Projects Releases Wiki Activity

how_to_show_ur_pp: fixes for end-2-end order/position display #60

Merged
goodboy merged 11 commits from how_to_show_ur_pp into main 2026-01-07 19:32:55 +00:00
Conversation 0 Commits 11 Files Changed 10
goodboy commented 2026-01-07 18:24:57 +00:00
Owner

Since apparently we can’t (yet) rely on our fqme (fully-qualified-market-endpoint)as consistent, e2e uniquely matchable keys for various brokerd -> emsd -> chartd event relaying, namely bc (at least from the .brokers.ib backend..) the venue-token-part can change depending on,

  • where the order was cleared, possibly on multiple (sub-)venues which are not the primary listing venue.
  • how cleared orders are recorded in the broker backend’s blotter/ledger normally by time but with a varying venue field.
  • depending on piker’s txn-ledger processing (again normally sorted by record time stamp) the last venue is likely not the primary; the fqme might not match that of the data feed’s MktPair.fqme.

This problems becomes most obvious with our chart-trading UX where position markers/annotations might not be displayed on the correct live time-series view due to such a mismatch of MktPair.fqme -> BrokerPosition.fmqe.

So instead (at least for now) just always relay through the broker’s own unique-mkt-ID schema (what we dub the bs_mktid: backend-sys market ID) on a new (5b91b089) BrokerdPosition.bs_mktid field to guarantee consistency at least on a per-broker reporting basis.

Obviously ensure we always set it for .ib emitted pp msgs (58654915) and ensure the .ui.order_mode prioritize the mkt-match on .bs_mktid (388a9a4d).

Related/surrounding broker txn ledger improvements

  • (b6d70d50, 7b68444c) track and ignore any txn entries with invalid time-stamps (again such as can be recorded by the .ib backend..) by refining the .accounting.calc.iter_by_dt() impl with a handy _invalid: list passed to the embedded dyn_parse_to_dt() closure which allows us to either,

    • crash-handle such cases when a new debug: bool param is set OR,
    • at the least log.error() such txns to console.

  • d67ace75 avoid overriding Account.pps: dict entries by again using a bs_mktid to uniquely key Position entries in .accounting._pos.open_account() as delivered by Account.pps.

  • 0e9b50de _ems: tolerate and warn on already popped execs,

    such that we never crash on a status_msg > = book._active.pop(oid) in the ‘closed’ status handler whenever a double removal happens.

  • c609858f ui._remote_ctl: shield remote rect removals

    Since under trio-cancellation the .remove() is a checkpoint

  • f5850fe5 draft a ems/txn-ledger test which needs to eventually have checks on an example ledger file ensuring a position cleared over multiple venues (and thus currently mapping over multiple .fqme keys in its txn set) renders to a single, unified and correct position measure/report.

Since apparently we can't (yet) rely on our fqme (fully-qualified-market-endpoint)as consistent, e2e uniquely matchable keys for various `brokerd` -> `emsd` -> `chartd` event relaying, namely bc (at least from the `.brokers.ib` backend..) the venue-token-part can change depending on, - where the order was cleared, possibly on multiple (sub-)venues which are **not the primary** listing venue. - how cleared orders are recorded in the broker backend's blotter/ledger normally by time but with a varying venue field. - depending on `piker`'s txn-ledger processing (again normally sorted by record time stamp) the last venue is likely not the *primary*; the fqme might not match that of the data feed's `MktPair.fqme`. This problems becomes most obvious with our chart-trading UX where position markers/annotations might not be displayed on the correct live time-series view due to such a mismatch of `MktPair.fqme` -> `BrokerPosition.fmqe`. So instead (at least for now) just always relay through the broker's own unique-mkt-ID schema (what we dub the `bs_mktid`: backend-sys market ID) on a new (5b91b089) `BrokerdPosition.bs_mktid` field to guarantee consistency at least on a per-broker reporting basis. Obviously ensure we always set it for `.ib` emitted pp msgs (58654915) and ensure the `.ui.order_mode` prioritize the mkt-match on `.bs_mktid` (388a9a4d). --- #### Related/surrounding broker txn ledger improvements - (b6d70d50, 7b68444c) track and ignore any txn entries with invalid time-stamps (again such as can be recorded by the `.ib` backend..) by refining the `.accounting.calc.iter_by_dt()` impl with a handy `_invalid: list` passed to the embedded `dyn_parse_to_dt()` closure which allows us to either, - crash-handle such cases when a new `debug: bool` param is set OR, - at the least `log.error()` such txns to console. - d67ace75 avoid overriding `Account.pps: dict` entries by again using a `bs_mktid` to uniquely key `Position` entries in `.accounting._pos.open_account()` as delivered by `Account.pps`. - 0e9b50de `_ems`: tolerate and warn on already popped execs, > such that we never crash on a `status_msg > = book._active.pop(oid)` > in the 'closed' status handler whenever a double removal happens. - c609858f `ui._remote_ctl`: shield remote rect removals > Since under `trio`-cancellation the `.remove()` is a checkpoint - f5850fe5 draft a ems/txn-ledger test which needs to eventually have checks on an example ledger file ensuring a position cleared over multiple venues (and thus currently mapping over multiple `.fqme` keys in its txn set) renders to a single, unified and correct position measure/report. * hopefully i won't forget about actually finishing this suite in follow-up with https://www.pikers.dev/pikers/piker/issues/61 !!
goodboy added 9 commits 2026-01-07 18:24:58 +00:00
4c3a51a326 ib-related: cope with invalid txn timestamps 
That is inside embedded `.accounting.calc.dyn_parse_to_dt()` closure add
an optional `_invalid: list` param to where we can report
bad-timestamped records which we instead override and return as
`from_timestamp(0.)` (when the parser loop falls through) and report
later (in summary ) from the `.accounting.calc.iter_by_dt()` caller. Add
some logging and an optional debug block for future tracing.

NOTE, this commit was re-edited during a conflict between the orig
branches: `dev/binance_api_3.1` & `dev/alt_tpts_for_perf`.
1f727c2b64 Don't override `Account.pps: dict` entries.. 
Despite a `.bs_mktid` ideally being a bijection with `MktPair.fqme`
values, apparently some backends (cough IB) will switch the .<venue>`
part in txn records resulting in multiple account-conf-file sections for
the same dst asset. Obviously that means we can't allocate new
`Position` entries keyed by that `bs_mktid`, instead be sure to **update
them instead**!

Deats,
- add case logic to avoid pp overwrites using a `pp_objs.get()` check.
- warn on duplicated pos entries whenever the current account-file
  entry's `mkt` doesn't match the pre-existing position's.
- mk `Position.add_clear()` return a `bool` indicating if the record was
  newly added, warn when it was already existing/added prior.

Also,
- drop the already deprecated `open_pps()`, also from sub-pkg exports.
- draft TODO for `Position.summary()` idea as a replacement for
  `BrokerdPosition`-msgs.
e4ec65c344 Add an option `BrokerdPosition.bs_mktid` field 
Such that backends can deliver their own internal unique
`MktPair.bs_mktid` when they can't seem to get it right via the
`.fqme: str` export.. (COUGH ib, you piece of sh#$).

Also add todo for possibly replacing the msg with a `Position.summary()`
"snapshot" as a better and more rigorously generated wire-ready msg.
54d1ef8273 ui.order_mode: prioritize mkt-match on `.bs_mktid` 
For backends which opt to set the new `BrokerdPosition.bs_mktid` field,
give (matching logic) priority to it such that even if the `.symbol`
field doesn't match the mkt currently focussed on chart, it will
always match on a provider's own internal asset-mapping-id. The original
fallback logic for `.fqme` matching is left as is.

As an example with IB, a qqq.nasdaq.ib txn may have been filled on
a non-primary venue as qqq.directedea.ib, in this case if the mkt is
displayed and focused on chart we want the **entire position info** to
be overlayed by the `OrderMode` UX without discrepancy.

Other refinements,
- improve logging and add a detailed edge-case-comment around the
  `.on_fill()` handler to clarify where if a benign 'error' msg is
  relayed from a backend it will cause the UI to operate as though the
  order **was not-cleared/cancelled** since the `.on_cancel()` handler
  will have likely been called just before, popping the `.dialogs`
  entry. Return `bool` to indicate whether the UI removed-lines
  / added-fill-arrows.
- inverse the `return` branching logic in `.on_cancel()` to reduce
  indent.
- add a very loud `log.error()` in `Status(resp='error')` case-block
  ensuring the console yells about the order being cancelled, also
  a todo for the weird msg-field recursion nonsense..
858f67b2bd `_ems`: tolerate and warn on already popped execs 
In the `translate_and_relay_brokerd_events()` loop task that is, such
that we never crash on a `status_msg = book._active.pop(oid)` in the
'closed' status handler whenever a double removal happens.

Turns out there were unforeseen races here when a benign backend error
would cause an order-mode dialog to be cancelled (incorrectly) and then
a UI side `.on_cancel()` would trigger too-early removal from the
`book._active` table despite the backend sending an actual 'closed'
event (much) later, this would crash on the now missing entry..

So instead we now,
- obviously use `book._active.pop(oid, None)`
- emit a `log.warning()` (not info lol) on a null-read and with a less
  "one-line-y" message explaining the double removal and maybe *why*.
344444cd7b `ui._remote_ctl`: shield remote rect removals 
Since under `trio`-cancellation the `.remove()` is a checkpoint and will
be masked by a taskc AND we **always want to remove the rect** despite
the surrounding teardown conditions.
1807b8826e Draft a gt-one-`.fqme`-in-txns/account-file test 
To start this is just a shell for the test, there's no checking logic
yet.. put it as `test_accounting.test_ib_account_with_duplicated_mktids()`.
The test is composed for now to be completely runtime-free using only
the offline txn-ledger / symcache / account loading APIs, ideally we
fill in the activated symbology-data-runtime cases once we figure a sane
way to handle incremental symcache updates for backends like IB..

To actually fill the test out with real checks we still need to,
- extract the problem account file from my ib.algopape into the test
  harness data.
- pick some contracts with multiple fqmes despite a single bs_mktid and
  ensure they're aggregated as a single `Position` as well as,
  * ideally de-duplicating txns from the account file section for the
    mkt..
  * warning appropriately about greater-then-one fqme for the bs_mktid
    and providing a way for the ledger re-writing to choose the
    appropriate `<venue>` as the "primary" when the
    data-symbology-runtime is up and possibly use it to incrementally
    update the IB symcache and store offline for next use?
goodboy force-pushed how_to_show_ur_pp from 7c11bdb859 to 58654915ac 2026-01-07 18:41:28 +00:00 Compare
goodboy added 1 commit 2026-01-07 19:22:04 +00:00
7b68444c7a accounting.calc: `.error()` on bad txn-time fields.. 
Since i'm seeing IB records with a `None` value and i don't want to be
debugging every time order-mode boots up..

Also use `pdb=debug` in `.open_ledger_dfs()`

Note, this had conflicts on `piker/accounting/calc.py` when rebasing
onto the refactored `brokers_refinery` history which were resolved
manually!
goodboy merged commit 9e82a46c0b into main 2026-01-07 19:32:55 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pikers/piker#60
There is no content yet.
Delete Branch "how_to_show_ur_pp"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?