From 99fb9d2eabd089b9fda2a0a84b410af28afd3727 Mon Sep 17 00:00:00 2001 From: goodboy Date: Tue, 21 Jul 2026 19:52:43 -0400 Subject: [PATCH] Plan gap-checker and backfiller repairs Lay out a correctness-first path from deterministic regressions through normalized gap detection, structured backfill completion, and crash-safe `NativeDB` persistence. Deats, - separate checker, storage, provider, and annotation gates - record confirmed current defects and data invariants - define commit boundaries and timeout-wrapped verification Prompt-IO: ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md (this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`)) --- .../20260721T215055Z_c6ec3d41_prompt_io.md | 41 ++ ...20260721T215055Z_c6ec3d41_prompt_io.raw.md | 26 ++ .../opencode/gap-checker-backfiller-repair.md | 436 ++++++++++++++++++ 3 files changed, 503 insertions(+) create mode 100644 ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md create mode 100644 ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md create mode 100644 plans/opencode/gap-checker-backfiller-repair.md diff --git a/ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md b/ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md new file mode 100644 index 00000000..1480ba97 --- /dev/null +++ b/ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md @@ -0,0 +1,41 @@ +--- +model: gpt-5.6-sol +provider: openai +service: opencode +session: ses_0799212ebffe42arY96czXn89F +timestamp: 2026-07-21T21:50:55Z +git_ref: c6ec3d41 +scope: docs +substantive: true +raw_file: 20260721T215055Z_c6ec3d41_prompt_io.raw.md +--- + +## Prompt + +The user asked to boot a plan to continue work on the +time-series gap checker and bug fixing of the history +backfiller. The user authorized prompt-I/O capture for the +plan documentation after the repository audit. + +## Response summary + +Produced a bounded repair plan based on current source, +historical issue caches, existing profiling notes, and a +test-coverage audit. The plan prioritizes reproducible +correctness failures and storage durability before venue +classification, annotation performance, or manual repair +UX. + +## Files changed + +- `plans/opencode/gap-checker-backfiller-repair.md` - + phased implementation and verification plan +- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md` + - unedited response record +- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md` + - provenance metadata and response summary + +## Human edits + +None - generated plan has not yet been edited by the +human. diff --git a/ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md b/ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md new file mode 100644 index 00000000..3e96618d --- /dev/null +++ b/ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md @@ -0,0 +1,26 @@ +--- +model: gpt-5.6-sol +provider: openai +service: opencode +timestamp: 2026-07-21T21:50:55Z +git_ref: c6ec3d41 +diff_cmd: git diff HEAD~1..HEAD +--- + +> `git diff -- plans/opencode/gap-checker-backfiller-repair.md` + +Generated a phased implementation plan for continuing +piker's gap checker and history backfiller repairs. The +plan separates data correctness from annotation +performance and orders work by dependency: deterministic +tests, canonical frame/gap normalization, structured +backfill completion, non-destructive atomic NativeDB +writes, exact null repair, provider/venue classification, +annotation vectorization, and crash qualification. + +The plan is grounded in current code paths, local issue +caches `#62`, `#71`, and `#75`, and the existing 1,285-gap +profiling handoff. It records confirmed present defects, +design invariants, per-phase acceptance gates, test +commands, commit boundaries, and explicitly deferred UX +and provider work. diff --git a/plans/opencode/gap-checker-backfiller-repair.md b/plans/opencode/gap-checker-backfiller-repair.md new file mode 100644 index 00000000..54a50eb0 --- /dev/null +++ b/plans/opencode/gap-checker-backfiller-repair.md @@ -0,0 +1,436 @@ +# Gap Checker and Backfiller Repair + +## Purpose + +Continue the time-series gap checker and backfiller work +from PRs/issues `#62`, `#71`, and `#75`, prioritizing data +correctness and crash safety before annotation UX or +rendering performance. + +Source baseline: + +- branch: `flake_update` +- commit: `c6ec3d41` +- gap handoff: + `notes_to_self/claudy/gapsannotator_sesh.md` +- primary code: + `piker/tsp/_anal.py`, `piker/tsp/_history.py`, + `piker/storage/nativedb.py`, and + `piker/tsp/_annotate.py` + +## Goals + +1. Make gap detection operate on sorted, unique, + validated OHLCV frames with fresh derived columns. +2. Make backfill completion, frame boundaries, and null + repair deterministic under exhaustion and cancellation. +3. Prevent a bounded or transient SHM view from + destructively replacing older persisted history. +4. Persist normalized history through crash-safe atomic + file replacement. +5. Distinguish expected venue closures from suspicious + missing data without production breakpoints. +6. Generate correct annotation specs before optimizing + the remaining client-side Polars bottleneck. + +## Confirmed Baseline Defects + +These are current-code findings to characterize in tests, +not assumptions inferred from historical task state. + +### Gap normalization + +- `dedupe()` computes `time_prev`, `s_diff`, and + `dt_diff` before sorting/deduplication, then returns + stale derived columns in the normalized frame + (`piker/tsp/_anal.py:683-729`). +- Negative/out-of-order deltas are treated as ordinary + gaps because `detect_time_gaps()` filters on + `abs(s_diff)` (`piker/tsp/_anal.py:582-638`). +- `get_null_segs()` drops single zero rows and its Polars + path does not define all state used by the shared path + (`piker/tsp/_anal.py:227-471`). +- `iter_null_segs()` ignores a supplied `null_segs` + result and rescans the frame (`piker/tsp/_anal.py:474-546`). + +### Backfill control and boundaries + +- `DataUnavailable` returns before `bf_done.set()`, while + `tsdb_backfill()` later waits forever on that event + (`piker/tsp/_history.py:516-536,1193-1212,1302-1308`). +- Empty arrays are indexed before the nominal null-frame + branch can handle them (`piker/tsp/_history.py:538-601`). +- Provider frame seams have no explicit inclusive/exclusive + contract, allowing duplicate adjacent timestamps in SHM. +- Cancellation may occur after SHM mutation but before + cursor and persistence bookkeeping + (`piker/tsp/_history.py:661-789`). + +### Null repair + +- Null query length is derived from an absolute endpoint, + not the null-span width (`piker/tsp/_history.py:278-287`). +- Fallback repair includes and mutates the valid leading + boundary row (`piker/tsp/_history.py:345-367`). +- Null repair and reverse backfill share one provider + client concurrently without enforcing the backend's + declared request limits. + +### Persistence + +- Every iteration passes bounded `shm.array` to + `write_ohlcv()`, while NativeDB replaces the complete + parquet file. Older rows that do not fit in SHM can be + lost (`piker/tsp/_history.py:722-789`, + `piker/storage/nativedb.py:305-368`). +- NativeDB writes directly to the final path and updates + its cache before the write succeeds; interruption can + leave disk and cache partial or inconsistent. +- Intentional zero-filled SHM space may be persisted while + null repair and reverse backfill run concurrently. + +### Annotation contract + +- `markup_gaps()` performs a full Polars filter per gap and + remains the measured client bottleneck + (`piker/tsp/_annotate.py:139-217`). +- Its missing-prior-row fallback selects the current row, + and `copysign(1, 0)` makes the flat-price branch + unreachable (`piker/tsp/_annotate.py:166-239`). +- Remote/reposition timestamp lookups index + `searchsorted()` results before excluding values equal + to `len(array)`. + +## Design Invariants + +All implementation phases preserve these rules: + +1. Persisted provider rows have nonzero, finite timestamps. +2. Normalized timestamps are strictly increasing and + unique. +3. Time-derived columns are computed only after final + sorting and duplicate resolution. +4. A detected gap explicitly carries its left and right + endpoints; consumers do not infer the left row from + `index - 1`. +5. Provider frame metadata equals the normalized array's + actual first and last timestamps. +6. Internal middleware uses one documented boundary + convention; provider adapters translate into it. +7. Expected sparse venue closures are represented by + timestamp distance, not persisted zero or synthetic + bars. +8. Storage merges preserve all unaffected older rows. +9. A storage commit becomes visible atomically, and cache + state changes only after disk commit succeeds. +10. Rendering or annotation failure cannot change the + checker or persistence result. + +## Phase 0: Deterministic Characterization + +Add synthetic OHLCV fixtures based on +`piker.data._source.def_iohlcv_fields`; avoid brokers, +networking, real actor trees, and Qt in the core suite. + +### Analysis tests + +Create `tests/test_tsp_analysis.py` covering: + +- uniform 1-second and 60-second series; +- one and multiple positive gaps; +- duplicate plus out-of-order timestamps; +- stale derived columns after sort/dedupe; +- zero, one, and multiple null segments; +- nulls at frame boundaries and nonzero absolute indexes; +- NumPy and Polars inputs; +- smart-dedupe conflict cases, including differing closes. + +### Backfill tests + +Create `tests/test_history_backfill.py` with scripted +`get_hist`, fake SHM, fake sampler stream, and recording +storage implementations. Characterize: + +- `DataUnavailable` completion without a hang; +- a genuinely empty frame; +- exact adjacency, overlap, and a positive storage gap; +- inclusive provider seam duplication; +- one-row and two-row startup history; +- cancellation after query, after SHM mutation, and + before persistence; +- one and multiple null rows with boundary preservation. + +### Storage tests + +Create `tests/test_storage_nativedb.py` using `tmp_path`: + +- NumPy and Polars round trips; +- preservation of history older than the SHM window; +- duplicate conflict policy; +- write failure leaves the prior parquet readable; +- cache changes only after a successful commit; +- separate FQME/timeframe series do not interfere. + +### Annotation tests + +Create `tests/test_gap_annotations.py` around a fake +`AnnotCtl`, without Qt initially: + +- left-close/right-open rectangle endpoints; +- up, down, and flat price direction; +- first-row and missing-prior-row behavior; +- empty gaps and out-of-range timestamps; +- deterministic batched rect/arrow/text payloads. + +Gate: every confirmed defect above has a focused failing +test or a written reason it requires a later integration +test. + +## Phase 1: Canonical Frame and Gap Model + +Refactor `piker/tsp/_anal.py` around one normalization +order: + +1. validate required columns and finite timestamps; +2. sort by `time`; +3. resolve duplicate timestamps under an explicit policy; +4. recompute all datetime/delta columns with `with_dts()`; +5. reject/report non-positive ordering deltas; +6. detect positive sample-step gaps. + +Keep the first implementation small: a normalized Polars +frame may remain the contract, but each gap row must carry +explicit `time_prev`, `index_prev`, `close_prev`, current +`time/index/open`, duration, and missing-sample count. + +Rewrite null grouping from extracted index/time arrays so +the NumPy and Polars paths share one implementation. Treat +a single null row as a normal one-row segment and honor a +precomputed `null_segs` argument without rescanning. + +Gate: analysis tests pass and no deduped/sorted frame +retains stale delta columns. + +## Phase 2: Structured Backfill Completion + +Refactor `start_backfill()` and `tsdb_backfill()` before +changing persistence semantics: + +- remove completion events that are awaited only after the + owning nursery has already joined its children, or make + completion a guaranteed `finally` outcome; +- represent `completed`, `provider_exhausted`, + `cancelled`, and `failed` as explicit results; +- check empty arrays before indexing endpoints; +- normalize each provider frame and recompute endpoint + metadata from the normalized data; +- define and enforce a disjoint seam between each new + reverse frame and the already-published frame; +- update cursor state immediately with the SHM mutation, + before any optional notification checkpoint; +- serialize reverse backfill and null repair unless a + backend capability explicitly permits concurrent + history requests; +- enforce declared backend request/rate limits. + +Prefer structured task return values over actor-local +events whose only consumer is the parent task. + +Gate: exhaustion and cancellation tests terminate within +their timeout, and SHM timestamps remain strictly +increasing across all frame seams. + +## Phase 3: Non-Destructive Atomic NativeDB Writes + +Stop treating the bounded SHM view as the full database. + +Add a NativeDB merge/update operation that: + +- accepts only the normalized provider frame being + committed; +- loads or uses the cached persisted frame; +- merges by timestamp while preserving unaffected rows; +- uses the historical provider bar as authoritative at a + closed-bar seam; +- validates ordering, uniqueness, schema, and nonzero + timestamps before writing; +- writes to a same-directory temporary parquet; +- reopens and validates the temporary file; +- atomically replaces the final path; +- updates `_dfs` only after replacement succeeds; +- serializes writers per `(fqme, timeframe)`. + +Backfill commits the validated frame/delta, never a live +`shm.array` snapshot containing mutable current bars or +intentional zero space. + +Gate: a persisted series can only grow backward or replace +explicitly overlapping timestamps; its earliest timestamp +never moves forward during ordinary backfill. + +## Phase 4: Exact Null Repair + +Use the canonical null spans from Phase 1: + +- query using explicit boundary timestamps; +- trim provider responses to the actual missing interval; +- write only null rows, never either valid boundary row; +- validate repaired timestamps and OHLC fields before + publication or persistence; +- leave expected venue closures sparse; +- do not persist synthetic flat bars without a durable + quality marker and an explicit schema decision. + +If provider data is unavailable, retain and report the +missing interval instead of silently converting it into +apparently valid history. + +Gate: single and multi-row repairs preserve both boundary +bars and either eliminate exactly the target nulls or +return an unresolved-gap result. + +## Phase 5: Venue and Provider Classification + +Apply gap classification after canonical detection rather +than embedding it in individual warning paths. + +For IB: + +- correct expiry comparison and weekday constants; +- classify internal positive gaps as well as trailing + request-coverage gaps; +- require closure interval alignment, not merely the + presence of a weekend/holiday somewhere in the gap; +- cover overnight sessions, weekends, holidays, DST, + active/expired futures, and unknown calendars; +- replace production `breakpoint()` calls with structured + data-quality results/errors. + +Provider adapters must return normalized endpoint metadata +and translate their native inclusive/exclusive APIs into +the middleware boundary contract. + +Gate: every gap is classified as expected closure, +suspicious missing data, provider exhaustion, or invalid +ordering; no production path pauses interactively. + +## Phase 6: Annotation Correctness and Performance + +Extract a synchronous, Qt-free spec builder from +`markup_gaps()`: + +```python +build_gap_annotation_specs(frame, gaps) +``` + +It consumes the explicit gap endpoints from Phase 1 and: + +- extracts needed columns once; +- replaces per-gap `wdts.filter()` with a vectorized join + or indexed lookup; +- handles equal prices explicitly; +- preserves previous-close/current-open semantics; +- styles expected closures separately from suspicious + gaps; +- emits deterministic rect/arrow/text batches. + +Then fix remote lookup/reposition bounds by masking +`searchsorted()` results before indexing and enforce both +FQME and timeframe ownership during redraw. + +Benchmark the same 1,285-gap workload from the Claude +handoff. Treat these as local benchmark goals, not portable +CI limits: + +- client spec build under 20 ms; +- server preparation under 10 ms; +- one composite gap graphics item; +- no full-frame Polars filter per gap; +- total local creation near or below 150 ms. + +Gate: annotation contract tests pass before performance +numbers are accepted. + +## Phase 7: Crash and Provider Qualification + +Run deterministic tests first, then the historical manual +matrix from issue `#62` on disposable storage: + +- fresh full backfill; +- append to existing history; +- graceful Ctrl-C; +- network interruption; +- SIGTERM and SIGKILL during query and parquet commit; +- restart and revalidate after every interruption. + +Start with Binance/Kraken/Kucoin, then run IB only with an +available account and gateway. Record before/after earliest +timestamp, latest timestamp, row count, unique count, null +count, non-positive deltas, and parquet readability. + +## Verification Commands + +Use an outer process timeout because actor teardown has a +known second-runtime wedge: + +```bash +timeout -k 5 30 python -m pytest -q \ + tests/test_tsp_analysis.py \ + tests/test_storage_nativedb.py + +timeout -k 5 60 python -m pytest -q \ + tests/test_history_backfill.py + +timeout -k 5 60 python -m pytest -q \ + tests/test_gap_annotations.py + +CI=1 timeout -k 20 120 python -m pytest -q -rs \ + tests/test_feeds.py +``` + +Run live/network tests separately from deterministic tests. +Do not automate `piker store anal` or `piker store ldshm`: +both may pause interactively or mutate attached data. + +## Commit Boundaries + +Keep implementation reviewable in this order: + +1. synthetic fixtures and analysis regressions; +2. canonical normalization and null grouping; +3. backfill completion and frame-boundary fixes; +4. NativeDB merge and atomic persistence; +5. exact null repair; +6. IB/provider gap classification; +7. annotation contract fixes and vectorization; +8. qualification notes and documentation. + +Each boundary must leave deterministic tests green. Do not +mix renderer optimization into storage/backfill commits. + +## Deferred + +The following remain follow-ups until the correctness +gates above pass: + +- chart context-menu/manual piecewise refill UX; +- automatic checker integration into every live chart; +- compact columnar annotation IPC; +- persistent synthetic-bar quality metadata/schema; +- full Deribit history repair and broad provider cleanup; +- Marketstore behavior changes; +- long-term incremental parquet partitioning. + +## Source References + +- `gitea/62.md`: workaround-era backfill and manual test + history +- `gitea/71.md`: IB venue closure behavior and pending + account verification +- `gitea/75.md`: gap annotation, storage races, and + backfiller follow-ups +- `notes_to_self/claudy/gapsannotator_sesh.md`: current + annotation benchmark and client bottleneck +- https://github.com/pikers/piker/pull/486 +- https://github.com/pikers/piker/pull/446 +- https://github.com/pikers/piker/issues/536