pikers
/
piker
4
1
Fork 0
Code Issues 14 Pull Requests 13 Packages Projects Releases Wiki Activity

ib_venue_closures: gap detection for "legacy mkts" #71

Merged
goodboy merged 7 commits from ib_venue_closures into main 2026-02-24 17:31:14 +00:00
Conversation 0 Commits 7 Files Changed 8
goodboy commented 2026-02-22 20:31:07 +00:00
Owner

Extension to #62 but for the ib backend now with sane exchange calendar processing of expected valid venue closure time-gaps.

Add a new piker.brokers.ib.venues mod which defines a buncha new helpers to aid with this, namely a is_venue_closore() which we use in the .api.Client.bars() to guard against any actual invalid non-closure gaps.

venues

Hi-level deats,

  • add a new .ib.venues.is_venue_closure() which compares gaps against ContractDetails.tradingSessions() converted to pendulum.Intervals from which we then extract to .start/.end.time(): pendulum.Time values.

  • also ensure classify any gap spanning is considered valid by checking it via the exchange_calendars lib in a new .venues.has_holiday().

  • also, use the new .venues.is_venue_open(), factored from orig logic in .ib.feed.stream_quotes(), to determine whether to wait for a first live quote during live data bootup.

NOTE, any such invalid gaps-cases we now expect to breakpoint() inside Client.bars() - since it should never hit if GFI’s data is legit (which is should be).

SO, if you’re using the ib backend do watch out for this and report it!!

🙏

Also includes a few other ib specific (backfiller machinery) improvements/reorgs:

  • dropping (well, commenting for now) recursive bar requests on frame-duration mismatches where previously Client.bars() would be re-invoked (from within itself) to attempt a max duration of data retrieval,

    • this had various problems not only to do with gaps but also seemingly with the recursion causing multiple underlying asyncio requests.
    • if we were to resume this approach for wtv reason it likely makes more sense to conduct it at the piker.data/.tsp layers.

  • use of ContractDetails.timeZoneId for all dts/durations so any introspecting human can grok what datum is what vs. the chart’s time x-axis is showing.

  • move the Client.bars() default param start_dt: str (a value in ISO 8601 format, as required by the IB api) to a new _iso8601_epoch_in_est: str mod level var so it can be easily accessed outside the method (if ever).

  • moved the gpt5 written parse_trading_hours() routine generated by @guille into new .ib.venues mod as well.

  • uppercase Crypto.symbol for PAXOS contracts in .find_contracts(), though now we’re getting a weird new API error..

    • left in a todo-comment to look at this.. definitely something to ask ib_async about possibly which should be shortly in follow up to this PR, > Error 10299, reqId 141: Expected what to show is > AGGTRADES, please use that instead of TRADES., > contract: Crypto(conId=479624278, symbol=‘BTC’, > exchange=‘PAXOS’, currency=‘USD’, > localSymbol=‘BTC.USD’, tradingClass=‘BTC’)

Before this lands,

  • some testing by users with actual accounts/data connections to verify this (mostly) works 🙏
  • maybe port us to ib_async (closing #68)?
    • my wager is just do another follow up patch.
Extension to #62 but for the `ib` backend now with sane exchange calendar processing of expected **valid** venue closure time-gaps. Add a new [`piker.brokers.ib.venues`](venues) mod which defines a buncha new helpers to aid with this, namely a `is_venue_closore()` which we use in the `.api.Client.bars()` to guard against any actual invalid non-closure gaps. [venues](https://www.pikers.dev/pikers/piker/pulls/71/files#diff-3e6d65a858f8734fc6719e01f872452bee241009) Hi-level deats, - add a new `.ib.venues.is_venue_closure()` which compares gaps against `ContractDetails.tradingSessions()` converted to `pendulum.Interval`s from which we then extract to `.start/.end.time(): pendulum.Time` values. - also ensure classify any gap spanning is considered valid by checking it via the `exchange_calendars` lib in a new `.venues.has_holiday()`. - also, use the new `.venues.is_venue_open()`, factored from orig logic in `.ib.feed.stream_quotes()`, to determine whether to wait for a first live quote during live data bootup. NOTE, any such invalid gaps-cases we now expect to `breakpoint()` inside `Client.bars()` - since it should never hit if GFI's data is legit (which is should be). SO, if you're using the ib backend do watch out for this and report it!! :pray: --- Also includes a few other `ib` specific (backfiller machinery) improvements/reorgs: - dropping (well, commenting for now) recursive bar requests on frame-duration mismatches where previously `Client.bars()` would be re-invoked (from within itself) to attempt a max duration of data retrieval, * this had various problems not only to do with gaps but also seemingly with the recursion causing multiple underlying `asyncio` requests. * if we were to resume this approach for wtv reason it likely makes more sense to conduct it at the `piker.data/.tsp` layers. - use of `ContractDetails.timeZoneId` for all dts/durations so any introspecting human can grok what datum is what vs. the chart's time x-axis is showing. - move the `Client.bars()` default param `start_dt: str` (a value in ISO 8601 format, as required by the IB api) to a new `_iso8601_epoch_in_est: str` mod level var so it can be easily accessed outside the method (if ever). - moved the gpt5 written `parse_trading_hours()` routine generated by `@guille` into new `.ib.venues` mod as well. - uppercase `Crypto.symbol` for `PAXOS` contracts in `.find_contracts()`, though now we're getting a weird new API error.. * left in a todo-comment to look at this.. definitely something to ask `ib_async` about possibly which should be shortly in follow up to this PR, > Error 10299, reqId 141: Expected what to show is > AGGTRADES, please use that instead of TRADES., > contract: Crypto(conId=479624278, symbol='BTC', > exchange='PAXOS', currency='USD', > localSymbol='BTC.USD', tradingClass='BTC') --- #### Before this lands, - [ ] some testing by users with actual accounts/data connections to verify this (mostly) works 🙏 - [ ] maybe port us to `ib_async` (closing #68)? * my wager is just do another follow up patch.
goodboy added 7 commits 2026-02-22 20:31:08 +00:00
9110edd77b Add `.ib.venues` for mkt-venue-closure checkers 
Introduce set of helper-fns for detecting venue open/close status,
session start/end times, and related time-gap detection using
`pendulum`.

Deats,
- add `iter_sessions()` to yield `pendulum.Interval`s from
  a `ContractDetails` instance.
- add `is_venue_open()` to check if active at a given time.
- add `is_venue_closure()` to detect valid closure gaps.
- add `sesh_times()` to extract weekday-agnostic open/close times.
- add `has_weekend()` to check for Sat/Sun in interval.
- move in lowlevel `is_current_time_in_range()` for checking a
  datetime within a `sesh: pendulum.Interval`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
4f845f5010 Mv `parse_trading_hours()` from `._util` to `.venues` 
It was an AI-model draft that we can prolly toss but figured might as
well org it appropriately.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
4c30547128 Add venue-closure gap-detection in `.ib.api.Client.bars()` 
With all detection logic coming from our new `.ib.venues` helpers
allowing use to verify IB's OHLC bars frames don't contain unexpected
time-gaps.

`Client.bars()` new checking deats,
- add `is_venue_open()`, `has_weekend()`, `sesh_times()`, and
  `is_venue_closure()` checks when `last_dt < end_dt`
- always calc gap-period in local tz via `ContractDetails.timeZoneId`.
- log warnings on invalid non-closure gaps, debug on closures for now.
- change recursion case to just `log.error()` + `breakpoint()`; we might end
  up tossing it since i don't think i could ever get it to be reliable..
  * mask-out recursive `.bars()` call (likely unnecessary).
- flip `start_dt`/`end_dt` param defaults to `None` vs epoch `str`.
- update docstring to clarify no `start_dt` support by IB
- add mod level `_iso8601_epoch_in_est` const to keep track of orig
  param default value.
- add multiline style to return type-annot, type all `pendulum` objects.

Also,
- uppercase `Crypto.symbol` for PAXOS contracts in `.find_contracts()`,
  tho now we're getting a weird new API error i left in a todo-comment..

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
47d0cfa2ff Use `.ib.venues.is_venue_open()` in `.feed` 
In `.ib.feed.stream_quotes()` specifically that is since time-range
checking code was moved to the new sub-mod.

Deats,
- drop import of old `is_current_time_in_range()` from `._util`
- change `get_bars()` sig: `end_dt`/`start_dt` to `datetime|None`
- comment-out `breakpoint()` in `open_history_client()`

Styling,
- add multiline style to conditionals and tuple unpacks
- fix type annotation: `Contract|None` vs `Contract | None`
- fix backticks in comment: `ib_insync` vs `ib_async`

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
04c1d999f3 Adjust type annots in binance and IB symbol mods 
Namely, again switching `|`-union syntax to rm adjacent white space.

Also, flip to multiline style for threshold comparison in
`.binance.feed` and change gap-check threshold to `timeframe` (vs
a hardcoded `60`s) in the `get_ohlc()` closure.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
8e89943a9e Add holiday-gap detection via `exchange_calendars` 
Integrate `exchange_calendars` lib to detect market holidays in
gap-checking logic via new `.ib.venues.has_holiday()` helper!

The `.ib.venues` impl deats,
- add  a new `has_holiday()` using `xcals.get_calendar()` and friends
  for sanity checking a venue's holiday closure-gaps.
  * final holiday detection-check is basically,
   `(cash_gap := (next_open - prev_close)) > period`
- include `time_step_s` param to `is_venue_closure()` for boundary
  tolerance checks.
  * let's us expand closure-time checks to include `+/-time_step_s`
    "off-by-one-`timeframe`-sample" edge case ranges.
- add real docstring to `has_weekend()`.

In `.ib.api` refine usage for ^ changes,
- move `is_venue_open()` call + tz-convert outside gap check
- use a walrus to capture `has_closure_gap` from `is_venue_closure()`
- add a `not has_closure_gap` condition to the
  mismatched-duration/short-frame warning block to avoid needless warns.
- keep duration-based "short-frame" log as `.error()` but toss in a bp
  so (somone can) umask to figure out wtf is going on..
  * we should **never** really hit this path unless there's a valid bug
    or data issue with IB/GFIS!
  * keep recursion path masked-out just leave a `breakpoint()` for now.

Also some logger updates,
- import `get_logger()` from top-level `piker.log` vs `.ib._util` which
  was always kinda wrong..
- change `NonShittyIB._logger` to use `__name__` vs literal.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
goodboy force-pushed ib_venue_closures from 8e89943a9e to 341a584cea 2026-02-22 23:41:52 +00:00 Compare
goodboy force-pushed ib_venue_closures from 341a584cea to b2b180428b 2026-02-23 01:06:07 +00:00 Compare
goodboy force-pushed ib_venue_closures from b2b180428b to 50011d33ef 2026-02-23 03:13:16 +00:00 Compare
goodboy changed title from ib_venue_closures: gap detection for "legacy mkts" to ib_venue_closures: gap detection for "legacy mkts" 2026-02-23 03:42:10 +00:00
goodboy changed target branch from hist_backfill_fixes to fix_tractor_logging 2026-02-23 03:42:11 +00:00
goodboy force-pushed ib_venue_closures from 50011d33ef to d17e6ab5d9 2026-02-23 18:41:06 +00:00 Compare
goodboy changed title from ib_venue_closures: gap detection for "legacy mkts" to ib_venue_closures: gap detection for "legacy mkts" 2026-02-24 17:30:21 +00:00
goodboy changed target branch from fix_tractor_logging to main 2026-02-24 17:30:21 +00:00
goodboy requested review from dnks 2026-02-24 17:30:31 +00:00
goodboy merged commit 9a720f8e21 into main 2026-02-24 17:31:14 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
dnks
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pikers/piker#71
There is no content yet.
Delete Branch "ib_venue_closures"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?