pikers
/
piker
4
1
Fork 0
Code Issues 16 Pull Requests 15 Packages Projects Releases Wiki Activity

gap_annotator: super fast (time-)gap detection API #75

Open
goodboy wants to merge 25 commits from gap_annotator into ib_venue_closures
pull from: gap_annotator
merge into: pikers:ib_venue_closures
Conversation 0 Commits 25 Files Changed 24

25 Commits (main)

Author SHA1 Message Date
Gud Boi b5a33e1217 Exclude crypto futes from `without_src` sym key 
Extend the `col_sym_key` asset-type check in `start_backfill()`
to also exclude crypto-denominated futures (where `src` is
`'crypto_currency'` and `dst` is `'future'`) from the
`without_src=True` fqme path.

Also in `.brokers.binance` backend (it being the guilty culprit in the
discovery of this bug; and why i touched styling this code),

- reformat `make_sub()` fn sig to multiline style in
  `.binance.feed`.
- add backtick around `dict` in `make_sub()` docstring.
- reformat `or` conditionals to multiline style in
  `.binance.feed.get_mkt_info()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 22:13:06 -05:00
Gud Boi 796a831c6e Enable console log (from passed down `loglevel`) in `.tsp._history.manage_history()` 2026-03-06 09:26:34 -05:00
Gud Boi de81d1e905 Drop `Flume.feed`, it's unused yet causes import cycles.. 2026-03-06 09:26:34 -05:00
Gud Boi 170dd9794c Just warn on single-bar nulls instead of bping 
Replace the debug breakpoint with a warning-log when a single-bar
null-segment is detected in `get_null_segs()`. This lets the gap
analysis continue while still alerting about the anomaly.

Deats,
- extract the 3-bar window (before, null, after) and calculate
  a `gap: pendulum.Interval` for the warning msg.
- comment-out the old breakpoint block for optional debugging as needed.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:26:34 -05:00
Gud Boi 599c36aba6 Lul, drop long unused poetry lock file 2026-03-06 09:26:34 -05:00
Gud Boi f174f79a1a Pin `pg` at latest official `0.14.0` release 
Keep in masked GH sources lines for easy hackin against upstream
`master` branch when needed as well!
 2026-03-06 09:26:34 -05:00
Gud Boi 9b284c2256 .ui._editors: log multiline styling and re-leveling 2026-03-06 09:26:34 -05:00
Gud Boi 59f2d46a97 .ui._lines: drop unused graphics-item import 2026-03-06 09:26:34 -05:00
Gud Boi c1b1e99693 Add batch-submit API for gap annotations 
Introduce `AnnotCtl.add_batch()` and `serve_rc_annots()` batch
handler to submit 1000s of gaps in single IPC msg instead of
per-annot round-trips. Server builds `GapAnnotations` from specs
and handles vectorized timestamp-to-index lookups.

Deats,
- add `'cmd': 'batch'` handler in `serve_rc_annots()`
- vectorized timestamp lookup via `np.searchsorted()` + masking
- build `gap_specs: list[dict]` from rect+arrow specs client-side
- create single `GapAnnotations` item for all gaps server-side
- handle `GapAnnotations.reposition()` in redraw handler
- add profiling to batch path for perf measurement
- support optional individual arrows for A/B comparison

Also,
- refactor `markup_gaps()` to collect specs + single batch call
- add `no_qt_updates()` context mgr for batch render ops
- add profiling to annotation teardown path
- add `GapAnnotations` case to `rm_annot()` match block

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:26:34 -05:00
Gud Boi 24651d2326 Add a `GapAnnotations` path-renderer 
For a ~1000x perf gain says ol' claudy, our boi who wrote this entire
patch! Bo

Introduce `GapAnnotations` in `.ui._annotate` for batch-rendering gap
rects/arrows instead of individual `QGraphicsItem` instances. Uses
upstream's `pyqtgraph.Qt.internals.PrimitiveArray` for rects and
a `QPainterPath` for arrows. This API-replicates our prior annotator's
in view shape-graphics but now using (what we're dubbing)
"single-array-multiple-graphics" tech much like our `.ui._curve`
extensions to `pg` B)

Impl deats,
- batch draw ~1000 gaps in single paint call vs 1000 items
- arrows render in scene coords to maintain pixel size on zoom
- add vectorized timestamp-to-index lookup for repositioning
- cache bounding rect, rebuild on `reposition()` calls
- match `SelectRect` + `ArrowItem` visual style/colors
- skip reposition when timeframe doesn't match gap's period

Other,
- fix typo in `LevelMarker` docstring: "graphich" -> "graphic"
- reflow docstring in `qgo_draw_markers()` to 67 char limit

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:26:34 -05:00
Gud Boi 2d00bb1024 Add info log for shm processing in `ldshm` CLI cmd 
Log shm file name and detected period before null segment
processing to aid debugging.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:26:34 -05:00
Gud Boi dd40ad603f Bump to latest official `pyqtgraph` release 2026-03-06 09:26:34 -05:00
Gud Boi f2ace1b63b Use `ppfmt()` in `order_mode` since it's provided by `tractor` now 2026-03-06 09:22:36 -05:00
Gud Boi 9010f9c7ab Augment `.ib.symbols` search with more logging 
Refactor `open_symbol_search()` to use `partial()` for nursery task
spawning and add detailed query->results logging via `ppfmt()`.

Deats,
- change `extend_results()` to accept `target` callable +
  `pattern` + `**kwargs` and invoke inside, instead of receiving
  a pre-called awaitable; use `partial()` to pass args.
- add `ppfmt()` formatted logging of search query params and
  results including client class + method repr.
- change `print()` -> `log.exception()` for `Lagged` overrun.
- bump `upto=5` -> `upto=10` for `search_symbols()` call.

Also for styling,
- add type some missing type annots.
- add multiline style to `or` conditionals in pattern check.
- reformat log msgs to multiline style throughout.
- use `ppfmt()` for fuzzy match debug log.
- rename nursery `sn` -> `tn`.
- add TODO comment about `assert 0` hang.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:22:36 -05:00
Gud Boi 89a145113c Handle `str`-errors in `.ib.broker` trade events 
Add `isinstance()` dispatch for the `'error'` event case in
`deliver_trade_events()` to handle `ib_async` sometimes emitting plain
`str` error items instead of the previously expected `dict`.

Deats,
- add `isinstance(err, dict)` branch for the standard case with
  `error_code`, `reason`, and `reqid` fields.
- add `isinstance(err, str)` branch to parse error strings of the
  form `'[code 104] connection failed'` into `code` and `reason`.
- set `reqid: str = '<unknown>'` for string-form errors since
  there's no request ID available.
- update `err` type annot to `dict|str`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:22:36 -05:00
Gud Boi ec4db30cdc Handle valid null frames and 0-bar cases in backfill 
Add guards for empty-array and zero-bar-diff cases in the TSP backfill
loops to avoid crashes and allow graceful loop termination.

In `maybe_fill_null_segments()`,
- add `array.size == 0` guard in `maybe_fill_null_segments()` to detect
  valid (venue closure) gaps from the backend; add a warning + bp
  + break for this case.
- add TODO that we should likely be filling nulls with the close price
  for the gap's duration.

In `start_backfill()`,
- expand the "0 bars after diff" warning msg with
  `backfill_until_dt` and `end_dt_param` context.
- mask the  `await tractor.pause()` and add a `break` to avoid blocking
  the backfill loop.

(this commit msg was generated in some part by
[`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:22:36 -05:00
Gud Boi 2a394dba03 Warn instead of raise on `start_dt`-trimmed frames 
Downgrade the `start_dt`-trimming check in `open_history_client()`
from a `RuntimeError` raise to a warning log, allowing the caller
to still receive a (shorter) frame of bars (though we may need to still
specially handle such cases in the backfiller's biz logic layer).

Deats,
- add `trimmed_bars.size` guard to skip check on empty results.
- change condition to `>=` and log a warning with the short-frame
  size instead of raising.
- comment-out `raise RuntimeError` and breakpoint for future
  removal once confident.
- add docstring-style comment on `start_dt=` kwarg noting that
  `Client.bars()` doesn't truly support it (uses duration-style
  queries internally).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:22:36 -05:00
Gud Boi 19f16e1df3 Handle ambiguous futes contracts in `get_fute()` 
Use (the only available in `ib_async`) `returnAll=True` in
`qualifyContractsAsync()` calls within `get_fute()` and handle the case
where IB returns a list of ambiguous contract matches instead of
a single result.

Deats,
- add `returnAll=True` to both `ContFuture` and `Future`
  qualification calls.
- add `isinstance(con, list)` check after unpacking first result
  to detect ambiguous contract sets.
- log warning with input params and matched contracts when
  ambiguous.
- update return type annot to `Contract|list[Contract]`.

Also,
- handle list-of-contracts case in `find_contracts()` by unpacking
  `*contracts` into the `qualifyContractsAsync()` call.
- reformat `qualifyContractsAsync()` calls to multiline style.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:22:36 -05:00
Gud Boi 3adb0d8b9d Fall back to `Contract.exchange` in `has_holiday()` 
Use `con.exchange` as fallback when `con.primaryExchange` is empty
in `has_holiday()` to handle contracts like futures that don't
always set a `primaryExchange`.

Deats,
- extract `con: Contract` from `con_deats.contract` for reuse.
- use `con.primaryExchange or con.exchange` to ensure a valid
  exchange code is always passed to the calendar lookup.
- add `Contract` to `TYPE_CHECKING` imports.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-03-06 09:22:36 -05:00
Gud Boi ee09f519a9 Remap non-std IB exchange values 
Add exchange name translation in `.ib.venues.has_holiday()` to handle
non-standard exchange codes when looking up holiday gaps..

Deats,
- add an ad-hoc lookup dict to remap an IB `Contract.primaryExchange` val
  which doesn't exist in the `exchange_calendars`'s alias set.
- use `.get()` with fallback to map `exch` to new `std_exch` and pass
  that to `xcals.get_calendar()`.
- add the case i just caught, `'ARCA'` -> `'ARCX'` to the table when i loaded
  the `gld.arca.ib` market..

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-02-26 18:02:39 -05:00
Gud Boi 9247746a79 Handle unknown order statuses in `.ib.broker` 
Add fallback handling for unknown IB order status strings to
avoid crashes when IB returns unexpected status values.

Deats,
- add `'ValidationError': 'error'` mapping to `_statuses` dict.
- use `.get()` with `'error'` default instead of direct dict
  lookup for `status.status`.
- add `elif status_str == 'error'` block to log unknown status
  values.
- add type annots to `event_name` and `item` in
  `deliver_trade_events()` loop.

Also,
- reformat log msg in `deliver_trade_events()` to multiline.
- drop extra conditional in `if status_str == 'filled'` check.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-02-26 18:01:03 -05:00
Gud Boi 9e2af2838f Ah right, we import types from `eventkit` (now `aeventkit`).. 2026-02-24 16:06:42 -05:00
Gud Boi b1cb67d1bd Port `.ib` backend from `ib_insync` to `ib_async` 
Migrate the IB broker backend to use `ib_async` (the actively maintained
fork) instead of the now stale, original `ib_insync` lib.

Deats,
- update `pyproject.toml` dep: drop `ib-insync` pin, add
  `ib-async>=2.1.0`.
- update lock file with `ib-async` and its new `aeventkit` dep (which
  i guess replaces `eventkit`).
- obvi, change all `ib_insync` imports to `ib_async` across `.ib.*`.
- update docs and select internal comments referencing the original lib.

Also,
- drop unused `ledger_dict` init in `_flex_reports.load_flex_trades()`.
- fix union type annot style: `dict | None` -> `dict|None`.
- strip `.tzinfo` from `lastTimeStamp` in `normalize()` to avoid
  IPC codec issues with `ib_async`'s `timezone.utc` injection.
- pop `'defaults'` from ticker data dict in `normalize()` to avoid
  non-serializable `timezone` objects and warning-log in such
  cases.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
 2026-02-24 15:13:28 -05:00
Gud Boi 196422433c Capture `cons` in `Client.get_fute()` 
That is to be able to (eventually) introspect "ambiguous" contract sets
once we move to `ib_async` and its `returnAll: bool` now offered by
`IB.qualifyContractsAsync()`,

https://github.com/ib-api-reloaded/ib_async/blob/main/ib_async/ib.py#L2115

Also, tweak some type type annots to multline style in sibling mods.
 2026-02-24 12:31:43 -05:00
Gud Boi 9a720f8e21 Merge pull request 'ib_venue_closures: gap detection for "legacy mkts"' (#71) 
Reviewed-on: https://www.pikers.dev/pikers/piker/pulls/71
 2026-02-24 17:31:13 +00:00