hist_backfill_fixes: working-around (some) conc issues in the tsdb backfiller #62

goodboy · 2026-01-07T19:36:17Z

goodboy commented

2026-01-07 19:36:17 +00:00

A bunch of first-draft implementation and reorg work around some long lingering backfiller bugs/issues which seem to have been introduced as part of moving to our .parquet-file backed default/builting tsdb per, colloquially dubbed nativedb.

I haven’t gone into much detail here about the fairly large set of changes since the explanation(s) will come in the follow up #75. We are landing this now since there are some subtle backfiller fixes which seem to resolve the most obvious issues (using mostly workarounds for now, unfortunately).

The very quick hi-level splanation is,

moving .tsp.__init__ content -> .tsp._history
extending the remote-annotation API (.ui._remote_ctl) for charts to include more pyqtgraph-graphics-type parameter passing and more endpoints to support all of arrows, txt-labels and rects.
- related extentions to .ui._editors.ArrowEditor as needed.
tweaks to the .data._sharedmem API for offline usage from non-owning, non-actor script processes.
mv .storage.cli.markup_gaps() -> new .tsp._annotate.
add .tsp._annotate.humanize_duration() for time-gap txt labeling content.
add .tsp._dedupe_smart (written by claude based on my heuristic-logic) to better de-duplicate real-time written bars versus differing versions from provider history.
add new .ui._style.get_fonts() for retrieving the dpi-aware instances.
add an “offline REPL runner” script claude wrote for itself to be able to grok a human dev’s experience introspecting crashes from pdb.
- since apparently it can’t actually allow TTY takeover by such tools (and/or can’t sub-spawn one inside it’s process tree..)
- means we also add pexpect as a dev dep.
pin to latest xonshrelease for @goodboy’s prompt injection needs.

Before landing,

(@momo) test the various new gap detection logics added to the piker.tsp._history subsys (and friends) namely for,
- no obvious time gaps created for various crypto-cex backends (binance, kraken, kucoin) which should be verified by both doing,
  - full backfills-from-scratch (after clearing previously saved histories by fqme using piker store delete <fqme>)
  - backfills with append-to previously saved time-series where there already is a history file (you can check using piker store ls) where both piker chart and pikerd were cancelled “gracefully” (using the normal ctl-c from console) and/or when the host lost it’s network connection/crashed/was-purposely-killed by the sys (say using `kill -SIGKILL $(pgrep -f pikerd).

In follow through,

(#75) open PR for our new gap-annotator + remote-ctl API

Follow-up-todos from GH,

-> also now moved to #75

A bunch of first-draft implementation and reorg work around some long lingering backfiller bugs/issues which seem to have been introduced as part of moving to our `.parquet`-file backed default/builting tsdb per, colloquially dubbed `nativedb`. I haven't gone into much detail here about the fairly large set of changes since the explanation(s) will come in the follow up #75. We are landing this now since there are some subtle backfiller fixes which seem to resolve the most obvious issues (using mostly workarounds for now, unfortunately). The very quick hi-level splanation is, - moving `.tsp.__init__` content -> `.tsp._history` - extending the remote-annotation API (`.ui._remote_ctl`) for charts to include more `pyqtgraph`-graphics-type parameter passing and more endpoints to support all of arrows, txt-labels and rects. * related extentions to `.ui._editors.ArrowEditor` as needed. - tweaks to the `.data._sharedmem` API for offline usage from non-owning, non-actor script processes. - mv `.storage.cli.markup_gaps()` -> new `.tsp._annotate`. - add `.tsp._annotate.humanize_duration()` for time-gap txt labeling content. - add `.tsp._dedupe_smart` (written by `claude` based on my heuristic-logic) to better de-duplicate real-time written bars versus differing versions from provider history. - add new `.ui._style.get_fonts()` for retrieving the dpi-aware instances. - add an “offline REPL runner” script `claude` wrote for itself to be able to grok a human dev’s experience introspecting crashes from `pdb`. * since apparently it can’t actually allow TTY takeover by such tools (and/or can’t sub-spawn one inside it’s process tree..) * means we also add `pexpect` as a dev dep. - [x] pin to latest `xonsh `release for @goodboy's prompt injection needs. --- #### Before landing, - [x] (@momo) test the various new gap detection logics added to the `piker.tsp._history` subsys (and friends) namely for, - [X] no obvious time gaps created for various crypto-cex backends (binance, kraken, kucoin) which should be verified by both doing, * full backfills-from-scratch (after clearing previously saved histories by fqme using `piker store delete <fqme>`) * backfills with **append-to** previously saved time-series where there already is a history file (you can check using `piker store ls`) where both `piker chart` and `pikerd` were cancelled "gracefully" (using the normal ctl-c from console) and/or when the host lost it's network connection/crashed/was-purposely-killed by the sys (say using `kill -SIGKILL $(pgrep -f pikerd). --- #### In follow through, - [x] (#75) open PR for our new gap-annotator + remote-ctl API --- #### Follow-up-todos from GH, -> also now moved to #75

goodboy force-pushed hist_backfill_fixes from 79eb8a1684 to d147bfe8c4

2026-01-16 02:27:31 +00:00

Compare

goodboy force-pushed hist_backfill_fixes from 9ebb977731 to cd6bc105de

2026-01-22 03:39:11 +00:00

Compare

goodboy added 1 commit 2026-01-22 04:52:20 +00:00

ad299789db Mv `markup_gaps()` to new `.tsp._annotate` mod

goodboy added 2 commits 2026-01-26 03:18:56 +00:00

809ec6accb Arrow editor refinements in prep for gap checker

Namely exposing `ArrowEditor.add()` params to provide access to
coloring/transparency settings over the remote-ctl annotation API and
also adding a new `.remove_all()` to easily clear all arrows from
a single call. Also add `.remove()` compat methods to the other editors
(i.e. for lines, rects).

e77bec203d Add arrow indicators to time gaps

Such that they're easier to spot when zoomed out, a similar color to the
`RectItem`s and also remote-controlled via the `AnnotCtl` api.

Deats,
- request an arrow per gap from `markup_gaps()` using a new
  `.add_arrow()` meth, set the color, direction and alpha with
  position always as the `iend`/close of the last valid bar.
- extend the `.ui._remote_ctl` subys to support the above,
  * add a new `AnnotCtl.add_arrow()`.
  * add the service-side IPC endpoint for a 'cmd': 'ArrowEditor'.
- add a new `rm_annot()` helper to ensure the right graphics removal
  API is used by annotation type:
  * `pg.ArrowItem` looks up the `ArrowEditor` and calls `.remove(annot).
  * `pg.SelectRect` keeps with calling `.delete()`.
- global-ize an `_editors` table to enable the prior.
- add an explicit RTE for races on the chart-actor's `_dss` init.

goodboy added 1 commit 2026-01-26 16:45:00 +00:00

ff502b62bf .ui.order_mode: multiline import styling

goodboy added 4 commits 2026-01-27 19:16:34 +00:00

4081336bd3 Catch too-early ib hist frames

For now by REPLing them and raising an RTE inside `.ib.feed` as well as
tracing any such cases that make it (from other providers) up to the
`.tsp._history` layer during null-segment backfilling.

d0966e0363 Space gap rect-annots "between" start-end bars

d2e1d6ce91 Add break for single bar null segments

f39b362bc4 Add a couple cooler "cooler"/"muted" red and greens

goodboy added 1 commit 2026-01-27 19:19:12 +00:00

8701b517e7 Add `pexpect`, `xonsh`@github:main to deps

The former bc `claude` needs it for its new "offline" REPL simulator
script `snippets/claude_debug_helper.py` and pin to `xonsh` git mainline
to get the fancy new next cmd/suggestion prompt feats (which @goodboy is
using from `modden` already). Bump lock file to match.

Ah right, and for now while hackin pin to a local `tractor` Bp

goodboy added 5 commits 2026-01-28 01:52:05 +00:00

be28d083e4 Expose more `pg.ArrowItem` params thru annot-ctl API

fdea8556d7 Lul, woops compare against first-dt in `.ib.feed` bars frame..

848c8ae533 ib._util: ignore timeout-errs when crash-handling `pyvnc` connects

1776242413 .ib.feed: trim bars frame to `start_dt`

de5b1737b4 Add humanized duration labels to gap annotations

Introduce `humanize_duration()` helper in `.tsp._annotate` to
convert seconds to short human-readable format (d/h/m/s). Extend
annot-ctl API with `add_text()` method for placing `pg.TextItem`
labels on charts.

Also,
- add duration labels on RHS of gap arrows in `markup_gaps()`
- handle text item removal in `rm_annot()` match block
- expose `TextItem` cmd in `serve_rc_annots()` IPC handler
- use `hcolor()` for named-to-hex color conversion
- set anchor positioning for up vs down gaps

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 1 commit 2026-01-28 01:53:56 +00:00

1fb0fe3a04 Add `font_size` param to `AnnotCtl.add_text()` API

Expose font sizing control for `pg.TextItem` annotations thru the
annot-ctl API. Default to `_font.font.pixelSize() - 3` when no
size provided.

Also,
- thread `font_size` param thru IPC handler in `serve_rc_annots()`
- apply font via `QFont.setPixelSize()` on text item creation
- add `?TODO` note in `markup_gaps()` re using `conf.toml` value
- update `add_text()` docstring with font_size param desc

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 1 commit 2026-01-28 02:10:12 +00:00

4e3cd7f986 Drop decimal points for whole-number durations

Adjust `humanize_duration()` to show "3h" instead of "3.0h" when the
duration value is a whole number, making labels cleaner.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 1 commit 2026-01-28 04:52:19 +00:00

76f199df3b Add buffer capacity checks to backfill loop

Prevent `ValueError` from negative prepend index in
`start_backfill()` by checking buffer space before push
attempts. Truncate incoming frame if needed and stop gracefully
when buffer full.

Also,
- add pre-push capacity check with frame truncation logic
- stop backfill when `next_prepend_index <= 0`
- log warnings for capacity exceeded and buffer-full conditions

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 1 commit 2026-01-28 17:52:33 +00:00

51d109f7e7 Do time-based shm-index lookup for annots on server

Fix annotation misalignment during backfill by switching from
client-computed indices to server-side timestamp lookups against
current shm state. Store absolute coords on annotations and
reposition on viz redraws.

Lowlevel impl deats,
- add `time` param to `.add_arrow()`, `.add_text()`, `.add_rect()`
- lookup indices from shm via timestamp matching in IPC handlers
- force chart redraw before `markup_gaps()` annotation creation
- wrap IPC send/receive in `trio.fail_after(3)` for timeout when
  server fails to respond, likely hangs on no-case-match/error.
- cache `_meth`/`_kwargs` on rects, `_abs_x`/`_abs_y` on arrows
- auto-reposition all annotations after viz reset in redraw cmd

Also,
- handle `KeyError` for missing timeframes in chart lookup
- return `-1` aid on annotation creation failures (lol oh `claude`..)
- reconstruct rect positions from timestamps + BGM offset logic
- log repositioned annotation counts on viz redraw

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 1 commit 2026-01-28 19:45:18 +00:00

858cfce958 Relay annot creation failures with err-dict resps

Change annot-ctl APIs to return `None` on failure instead of invalid
`aid`s. Server now sends `{'error': msg}` dict on failures, client
match-blocks handle gracefully.

Also,
- update return types: `.add_rect()`, `.add_arrow()`, `.add_text()`
  now return `int|None`
- match on `{'error': str(msg)}` in client IPC receive blocks
- send error dicts from server on timestamp lookup failures
- add failure handling in `markup_gaps()` to skip bad rects

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 2 commits 2026-01-28 21:32:47 +00:00

88732a67d5 Add `get_fonts()` API and fix `.px_size` for non-Qt ctxs

Add a public `.ui._style.get_fonts()` helper to retrieve the
`_font[_small]: DpiAwareFont` singleton pair. Adjust
`DpiAwareFont.px_size` to return `conf.toml` value when Qt returns `-1`
(no active Qt app).

Also,
- raise `ValueError` with detailed msg if both Qt and a conf-lookup fail
- add some more type union whitespace cleanups: `int | None` -> `int|None`

(this commit-msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

3a515afccd Use `get_fonts()`, add `show_txt` flag to gap annots

Switch `.tsp._annotate.markup_gaps()` to use new
`.ui._style.get_fonts()` API for font size calc on client side and add
optional `show_txt: bool` flag to toggle gap duration labels (with
default `False`).

Also,
- replace `sgn` checks with named bools: `up_gap`, `down_gap`
- use `small_font.px_size - 1` for gap label font sizing
- wrap text creation in `if show_txt:` block
- update IPC handler to use `get_fonts()` vs direct `_font` import

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 5 commits 2026-01-30 19:53:15 +00:00

8718ad4874 .ib._util: ignore attr err on click-hack twm wakeups?

f11ab5f0aa For claude, ignore no runtime for offline shm reading

205058de21 Always overwrite tsdb duplicates found during backfill

Enable the previously commented-out dedupe-and-write logic in
`start_backfill()` to ensure tsdb stays clean of duplicate
entries.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

ec4e6ec742 ib.feed: drop legacy "quote-with-vlm" polling

Since now we explicitly check each mkt's venue hours now we don't need
this mega hacky "waiting on a quote with real vlm" stuff to determing
whether historical data should be loaded immediately. This approach also
had the added complexity that we needed to handle edge cases for tickers
(like xauusd.cmdty) which never have vlm.. so it's nice to be rid of it
all ;p

88353ffef8 Ignore single-zero-sample trace on no runtime..

goodboy added 2 commits 2026-01-30 20:40:10 +00:00

eb78437994 Rm unused import in `.ui._curve`

bac8317a4a Add `get_godw()` singleton getter for `GodWidget`

Expose `get_godw()` helper to retrieve the central `GodWidget`
instance from anywhere in the UI code. Set the singleton in
`_async_main()` on startup.

Also,
- add docstring to `run_qtractor()` explaining trio guest mode
- type annotate `instance: GodWidget` in `run_qtractor()`
- import reorg in `._app` for cleaner grouping
- whitespace cleanup: `Type | None` -> `Type|None` throughout
- fix bitwise-or alignment: `Flag | Other` -> `Flag|Other`

(this commit-msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 2 commits 2026-01-30 23:48:34 +00:00

d5edd3484f Clarify `register_with_sampler()` started type and vars

Markup `ctx.started()` type-sig as `set[int]`, rename binding var
`first` to `shm_periods` and add type hints for clarity on context mgr
unpacking.

Also,
- whitespace cleanup: `Type | None` -> `Type|None` throughout
- format long lines: `.setdefault()`, `await ctx.started()`
- fix backtick style in docstrings for consistency
- add placeholder TODO comment for `feed_is_live` check; it might be
  more rigorous to pass the syncing state down thru all this?

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

f73b981173 Only register shms w sampler when `feed_is_live`

Add timeout-gated wait for `feed_is_live: trio.Event` before passing shm
tokens to `open_sample_stream()`; skip registering shm-buffers with the
sampler if the feed doesn't "go live" within a new timeout.

The main motivation here is to avoid the sampler incrementing shm-array
bufs when the mkt-venue is closed so that a trailing "same price"
line/bars isn't updated/rendered in the chart's view when unnecessary.

Deats,
- add `wait_for_live_timeout: float = 0.5` param to `manage_history()`
- warn-log the fqme when timeout triggers
- add error log for invalid `frame_start_dt` comparisons to
  `maybe_fill_null_segments()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 1 commit 2026-01-30 23:51:03 +00:00

48493e50b0 .ib.feed: only set `feed_is_live` after first quote

Move `feed_is_live.set()` to after receiving the first valid
quote instead of setting early on venue-closed path. Prevents
sampler registration when no live data expected.

Also,
- drop redundant `.set()` call in quote iteration loop
- add TODO note about sleeping until venue opens vs forever
- init `first_quote: dict` early for consistency

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 1 commit 2026-01-31 00:21:44 +00:00

2d678e1582 Guard against `None` chart in `ArrowEditor.remove()`

Add null check for `linked.chart` before calling
`.plotItem.removeItem()` to prevent `AttributeError` when chart
is `None`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy referenced this pull request

2026-01-31 20:36:18 +00:00

Add `.claude/skills/*` files from gap-annotator perf sesh with ma boi #69

goodboy added 1 commit 2026-02-02 00:39:38 +00:00

6f8a361e80 Cleanups and doc tweaks to `.ui._fsp`

Expand read-race warning log for clarity, add TODO for reading
`tractor` transport config from `conf.toml`, and reflow docstring
in `open_vlm_displays()`.

Also,
- whitespace cleanup: `Type | None` -> `Type|None`
- clarify "Volume" -> "Vlm (volume)" in docstr

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy added 3 commits 2026-02-20 21:38:11 +00:00

959d04024b .tsp._history: add gap detection in backfill loop

Add frame-gap detection when `frame_last_dt < end_dt_param` to
warn about potential venue closures or missing data during the
backfill loop in `start_backfill()`.

Deats,
- add `frame_last_dt < end_dt_param` check after frame recv
- log warnings with EST-converted timestamps for clarity
- add `await tractor.pause()` for REPL-investigation on gaps
- add TODO comment about venue closure hour checking
- capture `_until_was_none` walrus var for null-check clarity
- add `last_time` assertion for `time[-1] == next_end_dt`
- rename `_daterr` to `nodata` with `_nodata` capture

Also,
- import `pendulum.timezone` and create `est` tz instance
- change `get_logger()` import from `.data._util` to `.log`
- add parens around `(next_prepend_index - ln) < 0` check

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

f1b27e9696 Replace assert with warn for no-gaps in `.storage.cli`

Change `assert aids` to a warning log when no history gaps are found
during `ldshm` gap detection; it is the **ideal case** OBVI. This avoids
crashing the CLI when gap detection finds no issues, which is actually
good news!

Bp

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

ed9c211b96 Adjust binance stale-bar detection to 2x tolerance

Change the stale-bar check in `.binance.feed` from `timeframe` to
`timeframe * 2` tolerance to avoid false-positive pauses when bars
are slightly delayed but still within acceptable bounds.

Styling,
- add walrus operator to capture `_time_step` for debugger
  inspection.
- add comment explaining the debug purpose of this check.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

goodboy requested review from momo 2026-02-20 21:47:59 +00:00

goodboy referenced this pull request

2026-02-22 20:31:07 +00:00

ib_venue_closures: gap detection for "legacy mkts" #71

goodboy added 1 commit 2026-02-22 23:40:58 +00:00

e937b60ed6 Flip `.tsp._history` logger to explicit mod-name (again)

goodboy force-pushed hist_backfill_fixes from e937b60ed6 to 62e9c74377

2026-02-23 01:06:43 +00:00

Compare

momo changed title from ~~hist_backfill_fixes: solving conc issues in the tsdb backfiller~~ to WIP: hist_backfill_fixes: solving conc issues in the tsdb backfiller

2026-02-23 02:04:24 +00:00

momo changed title from ~~WIP: hist_backfill_fixes: solving conc issues in the tsdb backfiller~~ to hist_backfill_fixes: solving conc issues in the tsdb backfiller

2026-02-23 02:04:45 +00:00

goodboy force-pushed hist_backfill_fixes from 62e9c74377 to a97f6c8dcf

2026-02-23 03:13:23 +00:00

Compare

momo approved these changes 2026-02-23 04:15:30 +00:00

momo left a comment

-> I think this one was expected, so i used github link.

Caused by: Distribution not found at: file:///home/momo/Downloads/tractor

-> also with this command i got some warnings for tractor: piker store ls . expected warnings. [piker]$ piker store ls btcusdt.spot.binance btc.usdtm.perp.binance

-> piker store delete btcusdt.spot.binance

[piker]$ piker store delete btcusdt.spot.binance ->Then when did this: piker chart btcusdt.spot.binance there was no time gaps.

-> ctrl c and [piker]$ kill -9 $(pgrep -f “piker chart”) both worked, i had this: [10] > /home/momo/Downloads/piker/piker/tsp/_history.py(209)maybe_fill_null_segments() -> import greenback (Pdb+)

i dont know if i forgot to install dev dependencies, or the gap detection logic found actual gaps and triggered the fill routine (which needs greenback) , it comes from : maybe_fill_null_segments() )

-> I think this one was expected, so i used github link. Caused by: Distribution not found at: file:///home/momo/Downloads/tractor -> also with this command i got some warnings for tractor: piker store ls . expected warnings. [piker]$ piker store ls btcusdt.spot.binance btc.usdtm.perp.binance -> piker store delete btcusdt.spot.binance [piker]$ piker store delete btcusdt.spot.binance ->Then when did this: piker chart btcusdt.spot.binance there was no time gaps. -> ctrl c and [piker]$ kill -9 $(pgrep -f "piker chart") both worked, i had this: [10] > /home/momo/Downloads/piker/piker/tsp/_history.py(209)maybe_fill_null_segments() -> import greenback (Pdb+) i dont know if i forgot to install dev dependencies, or the gap detection logic found actual gaps and triggered the fill routine (which needs greenback) , it comes from : maybe_fill_null_segments() )

goodboy added 1 commit 2026-02-23 04:38:59 +00:00

90fce9fcd4 Woops, use `piker_pin` from GH for `tractor`

Also, install the `'repl'` deps-group by default to ensure we get the
extras required by `tractor` for non-`trio` task debug REPLin..
Bump lock file to match.

goodboy commented

2026-02-23 14:45:28 +00:00

Poster

-> I think this one was expected, so i used github link.

Caused by: Distribution not found at: file:///home/momo/Downloads/tractor

Yup.. i just put up a commit to fix.. 🤦

-> import greenback (Pdb+)

i dont know if i forgot to install dev dependencies, or the gap detection logic found actual gaps and triggered the fill routine (which needs greenback)

I also just added 'repl' to the default deps groups in that last commit which should hopefully resolve this.

> -> I think this one was expected, so i used github link. > > Caused by: Distribution not found at: file:///home/momo/Downloads/tractor Yup.. i just put up a commit to fix.. 🤦 > -> import greenback > (Pdb+) > > i dont know if i forgot to install dev dependencies, or the gap detection logic found actual gaps and triggered the fill routine (which needs greenback) I also just added `'repl'` to the default deps groups in that last commit which should hopefully resolve this.

goodboy commented

2026-02-23 15:51:39 +00:00

Poster

Yeah I just did one more round of testing on both ib and binance and i think landing this definitely doesn’t make things worse XD

I’m going to finish/move a couple of the bullets above and then land this puppy.

Yeah I just did one more round of testing on both ib and binance and i think landing this definitely doesn't make things worse XD I'm going to finish/move a couple of the bullets above and then land this puppy.

goodboy referenced this pull request

2026-02-23 16:28:57 +00:00

gap_annotator: super fast (time-)gap detection API #75

goodboy reviewed 2026-02-23 16:36:35 +00:00

piker/tsp/_annotate.py

													
				@ -0,0 +1,306 @@

				# piker: trading gear for hackers

goodboy commented

2026-02-23 16:36:35 +00:00

Poster

Ahh yeah, prolly worth noting that though it’s not documented yet, this is a a new .tsp mod which provides a bunch of new functionality for annotating time-gaps via a remote API started a while back when i first started digging into time-gap issues in the backfiller during the nativedb first-draft.

I’m not going to go through everything in detail here since it’s all going to be much more refined and formalized in #75 (and follow up).

Ahh yeah, prolly worth noting that though it's not documented yet, this is a a new `.tsp` mod which provides a bunch of new functionality for annotating time-gaps via a remote API started a while back when i first started digging into time-gap issues in the backfiller during the `nativedb` first-draft. I'm not going to go through everything in detail here since it's all going to be much more refined and formalized in #75 (and follow up).

goodboy reviewed 2026-02-23 16:40:46 +00:00

snippets/claude_debug_helper.py

													
				@ -0,0 +1,256 @@

				#!/usr/bin/env python

goodboy commented

2026-02-23 16:40:46 +00:00

Poster

Ahh right and this is an “offline REPL runner” script claude wrote for itself to be able to grok a human dev’s experience introspecting crashes from pdb since apparently it can’t actually allow TTY takeover by such tools (and/or can’t sub-spawn one inside it’s process tree..)

This prolly should be better refined as well once we have claude more integrated in our workflow.

Ahh right and this is an "offline REPL runner" script `claude` wrote for itself to be able to grok a human dev's experience introspecting crashes from `pdb` since apparently it can't actually allow TTY takeover by such tools (and/or can't sub-spawn one inside it's process tree..) This prolly should be better refined as well once we have `claude` more integrated in our workflow.

goodboy reviewed 2026-02-23 16:43:08 +00:00

piker/tsp/__init__.py Outdated

													
				@ -28,1435 +28,25 @@ Historical TSP (time-series processing) lowlevel mgmt machinery and biz logic fo

				  stored offline (in a tsdb).

				'''

				from __future__ import annotations

goodboy commented

2026-02-23 16:43:08 +00:00

Poster

Note this large change is just moving content to a new .tsp._history.

Note this large change is just moving content to a new `.tsp._history`.

goodboy reviewed 2026-02-23 16:44:38 +00:00

piker/tsp/_dedupe_smart.py

													
				@ -0,0 +10,4 @@

				from ._anal import with_dts

				def dedupe_ohlcv_smart(

goodboy commented

2026-02-23 16:44:38 +00:00

Poster

Using a heuristic for which bar(s) are likely most correct given known race conditions around OHLCV sampling under real-time-write usage.

goodboy changed title from ~~hist_backfill_fixes: solving conc issues in the tsdb backfiller~~ to hist_backfill_fixes: working-around (some) conc issues in the tsdb backfiller

2026-02-23 16:59:25 +00:00

goodboy reviewed 2026-02-23 17:08:05 +00:00

pyproject.toml Outdated

													
				@ -194,0 +198,4 @@

				# to get fancy next-cmd/suggestion feats prior to 0.22.2 B)

				# https://github.com/xonsh/xonsh/pull/6037

				# https://github.com/xonsh/xonsh/pull/6048

				xonsh = { git = 'https://github.com/xonsh/xonsh.git', branch = 'main' }