pikers
/
piker
4
1
Fork 0
Code Issues 20 Pull Requests 11 Packages Projects Releases Wiki Activity

Plan gap-checker and backfiller repairs 

Lay out a correctness-first path from deterministic regressions
through normalized gap detection, structured backfill completion,
and crash-safe `NativeDB` persistence.

Deats,
- separate checker, storage, provider, and annotation gates
- record confirmed current defects and data invariants
- define commit boundaries and timeout-wrapped verification

Prompt-IO: ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
backfiller_deep_fixes
Gud Boi 2026-07-21 19:52:43 -04:00
parent e766c158de
commit 99fb9d2eab
3 changed files with 503 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md 100644
View File

@ -0,0 +1,41 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-21T21:50:55Z
git_ref: c6ec3d41
scope: docs
substantive: true
raw_file: 20260721T215055Z_c6ec3d41_prompt_io.raw.md
---
## Prompt
The user asked to boot a plan to continue work on the
time-series gap checker and bug fixing of the history
backfiller. The user authorized prompt-I/O capture for the
plan documentation after the repository audit.
## Response summary
Produced a bounded repair plan based on current source,
historical issue caches, existing profiling notes, and a
test-coverage audit. The plan prioritizes reproducible
correctness failures and storage durability before venue
classification, annotation performance, or manual repair
UX.
## Files changed
- `plans/opencode/gap-checker-backfiller-repair.md` -
phased implementation and verification plan
- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated plan has not yet been edited by the
human.

26
ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md 100644
View File

@ -0,0 +1,26 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-21T21:50:55Z
git_ref: c6ec3d41
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff -- plans/opencode/gap-checker-backfiller-repair.md`
Generated a phased implementation plan for continuing
piker's gap checker and history backfiller repairs. The
plan separates data correctness from annotation
performance and orders work by dependency: deterministic
tests, canonical frame/gap normalization, structured
backfill completion, non-destructive atomic NativeDB
writes, exact null repair, provider/venue classification,
annotation vectorization, and crash qualification.
The plan is grounded in current code paths, local issue
caches `#62`, `#71`, and `#75`, and the existing 1,285-gap
profiling handoff. It records confirmed present defects,
design invariants, per-phase acceptance gates, test
commands, commit boundaries, and explicitly deferred UX
and provider work.

436
plans/opencode/gap-checker-backfiller-repair.md 100644
View File

@ -0,0 +1,436 @@
# Gap Checker and Backfiller Repair
## Purpose
Continue the time-series gap checker and backfiller work
from PRs/issues `#62`, `#71`, and `#75`, prioritizing data
correctness and crash safety before annotation UX or
rendering performance.
Source baseline:
- branch: `flake_update`
- commit: `c6ec3d41`
- gap handoff:
`notes_to_self/claudy/gapsannotator_sesh.md`
- primary code:
`piker/tsp/_anal.py`, `piker/tsp/_history.py`,
`piker/storage/nativedb.py`, and
`piker/tsp/_annotate.py`
## Goals
1. Make gap detection operate on sorted, unique,
validated OHLCV frames with fresh derived columns.
2. Make backfill completion, frame boundaries, and null
repair deterministic under exhaustion and cancellation.
3. Prevent a bounded or transient SHM view from
destructively replacing older persisted history.
4. Persist normalized history through crash-safe atomic
file replacement.
5. Distinguish expected venue closures from suspicious
missing data without production breakpoints.
6. Generate correct annotation specs before optimizing
the remaining client-side Polars bottleneck.
## Confirmed Baseline Defects
These are current-code findings to characterize in tests,
not assumptions inferred from historical task state.
### Gap normalization
- `dedupe()` computes `time_prev`, `s_diff`, and
`dt_diff` before sorting/deduplication, then returns
stale derived columns in the normalized frame
(`piker/tsp/_anal.py:683-729`).
- Negative/out-of-order deltas are treated as ordinary
gaps because `detect_time_gaps()` filters on
`abs(s_diff)` (`piker/tsp/_anal.py:582-638`).
- `get_null_segs()` drops single zero rows and its Polars
path does not define all state used by the shared path
(`piker/tsp/_anal.py:227-471`).
- `iter_null_segs()` ignores a supplied `null_segs`
result and rescans the frame (`piker/tsp/_anal.py:474-546`).
### Backfill control and boundaries
- `DataUnavailable` returns before `bf_done.set()`, while
`tsdb_backfill()` later waits forever on that event
(`piker/tsp/_history.py:516-536,1193-1212,1302-1308`).
- Empty arrays are indexed before the nominal null-frame
branch can handle them (`piker/tsp/_history.py:538-601`).
- Provider frame seams have no explicit inclusive/exclusive
contract, allowing duplicate adjacent timestamps in SHM.
- Cancellation may occur after SHM mutation but before
cursor and persistence bookkeeping
(`piker/tsp/_history.py:661-789`).
### Null repair
- Null query length is derived from an absolute endpoint,
not the null-span width (`piker/tsp/_history.py:278-287`).
- Fallback repair includes and mutates the valid leading
boundary row (`piker/tsp/_history.py:345-367`).
- Null repair and reverse backfill share one provider
client concurrently without enforcing the backend's
declared request limits.
### Persistence
- Every iteration passes bounded `shm.array` to
`write_ohlcv()`, while NativeDB replaces the complete
parquet file. Older rows that do not fit in SHM can be
lost (`piker/tsp/_history.py:722-789`,
`piker/storage/nativedb.py:305-368`).
- NativeDB writes directly to the final path and updates
its cache before the write succeeds; interruption can
leave disk and cache partial or inconsistent.
- Intentional zero-filled SHM space may be persisted while
null repair and reverse backfill run concurrently.
### Annotation contract
- `markup_gaps()` performs a full Polars filter per gap and
remains the measured client bottleneck
(`piker/tsp/_annotate.py:139-217`).
- Its missing-prior-row fallback selects the current row,
and `copysign(1, 0)` makes the flat-price branch
unreachable (`piker/tsp/_annotate.py:166-239`).
- Remote/reposition timestamp lookups index
`searchsorted()` results before excluding values equal
to `len(array)`.
## Design Invariants
All implementation phases preserve these rules:
1. Persisted provider rows have nonzero, finite timestamps.
2. Normalized timestamps are strictly increasing and
unique.
3. Time-derived columns are computed only after final
sorting and duplicate resolution.
4. A detected gap explicitly carries its left and right
endpoints; consumers do not infer the left row from
`index - 1`.
5. Provider frame metadata equals the normalized array's
actual first and last timestamps.
6. Internal middleware uses one documented boundary
convention; provider adapters translate into it.
7. Expected sparse venue closures are represented by
timestamp distance, not persisted zero or synthetic
bars.
8. Storage merges preserve all unaffected older rows.
9. A storage commit becomes visible atomically, and cache
state changes only after disk commit succeeds.
10. Rendering or annotation failure cannot change the
checker or persistence result.
## Phase 0: Deterministic Characterization
Add synthetic OHLCV fixtures based on
`piker.data._source.def_iohlcv_fields`; avoid brokers,
networking, real actor trees, and Qt in the core suite.
### Analysis tests
Create `tests/test_tsp_analysis.py` covering:
- uniform 1-second and 60-second series;
- one and multiple positive gaps;
- duplicate plus out-of-order timestamps;
- stale derived columns after sort/dedupe;
- zero, one, and multiple null segments;
- nulls at frame boundaries and nonzero absolute indexes;
- NumPy and Polars inputs;
- smart-dedupe conflict cases, including differing closes.
### Backfill tests
Create `tests/test_history_backfill.py` with scripted
`get_hist`, fake SHM, fake sampler stream, and recording
storage implementations. Characterize:
- `DataUnavailable` completion without a hang;
- a genuinely empty frame;
- exact adjacency, overlap, and a positive storage gap;
- inclusive provider seam duplication;
- one-row and two-row startup history;
- cancellation after query, after SHM mutation, and
before persistence;
- one and multiple null rows with boundary preservation.
### Storage tests
Create `tests/test_storage_nativedb.py` using `tmp_path`:
- NumPy and Polars round trips;
- preservation of history older than the SHM window;
- duplicate conflict policy;
- write failure leaves the prior parquet readable;
- cache changes only after a successful commit;
- separate FQME/timeframe series do not interfere.
### Annotation tests
Create `tests/test_gap_annotations.py` around a fake
`AnnotCtl`, without Qt initially:
- left-close/right-open rectangle endpoints;
- up, down, and flat price direction;
- first-row and missing-prior-row behavior;
- empty gaps and out-of-range timestamps;
- deterministic batched rect/arrow/text payloads.
Gate: every confirmed defect above has a focused failing
test or a written reason it requires a later integration
test.
## Phase 1: Canonical Frame and Gap Model
Refactor `piker/tsp/_anal.py` around one normalization
order:
1. validate required columns and finite timestamps;
2. sort by `time`;
3. resolve duplicate timestamps under an explicit policy;
4. recompute all datetime/delta columns with `with_dts()`;
5. reject/report non-positive ordering deltas;
6. detect positive sample-step gaps.
Keep the first implementation small: a normalized Polars
frame may remain the contract, but each gap row must carry
explicit `time_prev`, `index_prev`, `close_prev`, current
`time/index/open`, duration, and missing-sample count.
Rewrite null grouping from extracted index/time arrays so
the NumPy and Polars paths share one implementation. Treat
a single null row as a normal one-row segment and honor a
precomputed `null_segs` argument without rescanning.
Gate: analysis tests pass and no deduped/sorted frame
retains stale delta columns.
## Phase 2: Structured Backfill Completion
Refactor `start_backfill()` and `tsdb_backfill()` before
changing persistence semantics:
- remove completion events that are awaited only after the
owning nursery has already joined its children, or make
completion a guaranteed `finally` outcome;
- represent `completed`, `provider_exhausted`,
`cancelled`, and `failed` as explicit results;
- check empty arrays before indexing endpoints;
- normalize each provider frame and recompute endpoint
metadata from the normalized data;
- define and enforce a disjoint seam between each new
reverse frame and the already-published frame;
- update cursor state immediately with the SHM mutation,
before any optional notification checkpoint;
- serialize reverse backfill and null repair unless a
backend capability explicitly permits concurrent
history requests;
- enforce declared backend request/rate limits.
Prefer structured task return values over actor-local
events whose only consumer is the parent task.
Gate: exhaustion and cancellation tests terminate within
their timeout, and SHM timestamps remain strictly
increasing across all frame seams.
## Phase 3: Non-Destructive Atomic NativeDB Writes
Stop treating the bounded SHM view as the full database.
Add a NativeDB merge/update operation that:
- accepts only the normalized provider frame being
committed;
- loads or uses the cached persisted frame;
- merges by timestamp while preserving unaffected rows;
- uses the historical provider bar as authoritative at a
closed-bar seam;
- validates ordering, uniqueness, schema, and nonzero
timestamps before writing;
- writes to a same-directory temporary parquet;
- reopens and validates the temporary file;
- atomically replaces the final path;
- updates `_dfs` only after replacement succeeds;
- serializes writers per `(fqme, timeframe)`.
Backfill commits the validated frame/delta, never a live
`shm.array` snapshot containing mutable current bars or
intentional zero space.
Gate: a persisted series can only grow backward or replace
explicitly overlapping timestamps; its earliest timestamp
never moves forward during ordinary backfill.
## Phase 4: Exact Null Repair
Use the canonical null spans from Phase 1:
- query using explicit boundary timestamps;
- trim provider responses to the actual missing interval;
- write only null rows, never either valid boundary row;
- validate repaired timestamps and OHLC fields before
publication or persistence;
- leave expected venue closures sparse;
- do not persist synthetic flat bars without a durable
quality marker and an explicit schema decision.
If provider data is unavailable, retain and report the
missing interval instead of silently converting it into
apparently valid history.
Gate: single and multi-row repairs preserve both boundary
bars and either eliminate exactly the target nulls or
return an unresolved-gap result.
## Phase 5: Venue and Provider Classification
Apply gap classification after canonical detection rather
than embedding it in individual warning paths.
For IB:
- correct expiry comparison and weekday constants;
- classify internal positive gaps as well as trailing
request-coverage gaps;
- require closure interval alignment, not merely the
presence of a weekend/holiday somewhere in the gap;
- cover overnight sessions, weekends, holidays, DST,
active/expired futures, and unknown calendars;
- replace production `breakpoint()` calls with structured
data-quality results/errors.
Provider adapters must return normalized endpoint metadata
and translate their native inclusive/exclusive APIs into
the middleware boundary contract.
Gate: every gap is classified as expected closure,
suspicious missing data, provider exhaustion, or invalid
ordering; no production path pauses interactively.
## Phase 6: Annotation Correctness and Performance
Extract a synchronous, Qt-free spec builder from
`markup_gaps()`:
```python
build_gap_annotation_specs(frame, gaps)
```
It consumes the explicit gap endpoints from Phase 1 and:
- extracts needed columns once;
- replaces per-gap `wdts.filter()` with a vectorized join
or indexed lookup;
- handles equal prices explicitly;
- preserves previous-close/current-open semantics;
- styles expected closures separately from suspicious
gaps;
- emits deterministic rect/arrow/text batches.
Then fix remote lookup/reposition bounds by masking
`searchsorted()` results before indexing and enforce both
FQME and timeframe ownership during redraw.
Benchmark the same 1,285-gap workload from the Claude
handoff. Treat these as local benchmark goals, not portable
CI limits:
- client spec build under 20 ms;
- server preparation under 10 ms;
- one composite gap graphics item;
- no full-frame Polars filter per gap;
- total local creation near or below 150 ms.
Gate: annotation contract tests pass before performance
numbers are accepted.
## Phase 7: Crash and Provider Qualification
Run deterministic tests first, then the historical manual
matrix from issue `#62` on disposable storage:
- fresh full backfill;
- append to existing history;
- graceful Ctrl-C;
- network interruption;
- SIGTERM and SIGKILL during query and parquet commit;
- restart and revalidate after every interruption.
Start with Binance/Kraken/Kucoin, then run IB only with an
available account and gateway. Record before/after earliest
timestamp, latest timestamp, row count, unique count, null
count, non-positive deltas, and parquet readability.
## Verification Commands
Use an outer process timeout because actor teardown has a
known second-runtime wedge:
```bash
timeout -k 5 30 python -m pytest -q \
tests/test_tsp_analysis.py \
tests/test_storage_nativedb.py
timeout -k 5 60 python -m pytest -q \
tests/test_history_backfill.py
timeout -k 5 60 python -m pytest -q \
tests/test_gap_annotations.py
CI=1 timeout -k 20 120 python -m pytest -q -rs \
tests/test_feeds.py
```
Run live/network tests separately from deterministic tests.
Do not automate `piker store anal` or `piker store ldshm`:
both may pause interactively or mutate attached data.
## Commit Boundaries
Keep implementation reviewable in this order:
1. synthetic fixtures and analysis regressions;
2. canonical normalization and null grouping;
3. backfill completion and frame-boundary fixes;
4. NativeDB merge and atomic persistence;
5. exact null repair;
6. IB/provider gap classification;
7. annotation contract fixes and vectorization;
8. qualification notes and documentation.
Each boundary must leave deterministic tests green. Do not
mix renderer optimization into storage/backfill commits.
## Deferred
The following remain follow-ups until the correctness
gates above pass:
- chart context-menu/manual piecewise refill UX;
- automatic checker integration into every live chart;
- compact columnar annotation IPC;
- persistent synthetic-bar quality metadata/schema;
- full Deribit history repair and broad provider cleanup;
- Marketstore behavior changes;
- long-term incremental parquet partitioning.
## Source References
- `gitea/62.md`: workaround-era backfill and manual test
history
- `gitea/71.md`: IB venue closure behavior and pending
account verification
- `gitea/75.md`: gap annotation, storage races, and
backfiller follow-ups
- `notes_to_self/claudy/gapsannotator_sesh.md`: current
annotation benchmark and client bottleneck
- https://github.com/pikers/piker/pull/486
- https://github.com/pikers/piker/pull/446
- https://github.com/pikers/piker/issues/536