piker/plans/opencode/gap-checker-backfiller-repair.md

Gap Checker and Backfiller Repair

Purpose

Continue the time-series gap checker and backfiller work from PRs/issues #62, #71, and #75, prioritizing data correctness and crash safety before annotation UX or rendering performance.

Source baseline:

• branch: flake_update
• commit: c6ec3d41
• gap handoff: notes_to_self/claudy/gapsannotator_sesh.md
• primary code: piker/tsp/_anal.py, piker/tsp/_history.py, piker/storage/nativedb.py, and piker/tsp/_annotate.py

Goals

1. Make gap detection operate on sorted, unique, validated OHLCV frames with fresh derived columns.
2. Make backfill completion, frame boundaries, and null repair deterministic under exhaustion and cancellation.
3. Prevent a bounded or transient SHM view from destructively replacing older persisted history.
4. Persist normalized history through crash-safe atomic file replacement.
5. Distinguish expected venue closures from suspicious missing data without production breakpoints.
6. Generate correct annotation specs before optimizing the remaining client-side Polars bottleneck.

Confirmed Baseline Defects

These are current-code findings to characterize in tests, not assumptions inferred from historical task state.

Gap normalization

• dedupe() computes time_prev, s_diff, and dt_diff before sorting/deduplication, then returns stale derived columns in the normalized frame (piker/tsp/_anal.py:683-729).
• Negative/out-of-order deltas are treated as ordinary gaps because detect_time_gaps() filters on abs(s_diff) (piker/tsp/_anal.py:582-638).
• get_null_segs() drops single zero rows and its Polars path does not define all state used by the shared path (piker/tsp/_anal.py:227-471).
• iter_null_segs() ignores a supplied null_segs result and rescans the frame (piker/tsp/_anal.py:474-546).

Backfill control and boundaries

• DataUnavailable returns before bf_done.set(), while tsdb_backfill() later waits forever on that event (piker/tsp/_history.py:516-536,1193-1212,1302-1308).
• Empty arrays are indexed before the nominal null-frame branch can handle them (piker/tsp/_history.py:538-601).
• Provider frame seams have no explicit inclusive/exclusive contract, allowing duplicate adjacent timestamps in SHM.
• Cancellation may occur after SHM mutation but before cursor and persistence bookkeeping (piker/tsp/_history.py:661-789).

Null repair

• Null query length is derived from an absolute endpoint, not the null-span width (piker/tsp/_history.py:278-287).
• Fallback repair includes and mutates the valid leading boundary row (piker/tsp/_history.py:345-367).
• Null repair and reverse backfill share one provider client concurrently without enforcing the backend’s declared request limits.

Persistence

• Every iteration passes bounded shm.array to write_ohlcv(), while NativeDB replaces the complete parquet file. Older rows that do not fit in SHM can be lost (piker/tsp/_history.py:722-789, piker/storage/nativedb.py:305-368).
• NativeDB writes directly to the final path and updates its cache before the write succeeds; interruption can leave disk and cache partial or inconsistent.
• Intentional zero-filled SHM space may be persisted while null repair and reverse backfill run concurrently.

Annotation contract

• markup_gaps() performs a full Polars filter per gap and remains the measured client bottleneck (piker/tsp/_annotate.py:139-217).
• Its missing-prior-row fallback selects the current row, and copysign(1, 0) makes the flat-price branch unreachable (piker/tsp/_annotate.py:166-239).
• Remote/reposition timestamp lookups index searchsorted() results before excluding values equal to len(array).

Design Invariants

All implementation phases preserve these rules:

1. Persisted provider rows have nonzero, finite timestamps.
2. Normalized timestamps are strictly increasing and unique.
3. Time-derived columns are computed only after final sorting and duplicate resolution.
4. A detected gap explicitly carries its left and right endpoints; consumers do not infer the left row from index - 1.
5. Provider frame metadata equals the normalized array’s actual first and last timestamps.
6. Internal middleware uses one documented boundary convention; provider adapters translate into it.
7. Expected sparse venue closures are represented by timestamp distance, not persisted zero or synthetic bars.
8. Storage merges preserve all unaffected older rows.
9. A storage commit becomes visible atomically, and cache state changes only after disk commit succeeds.
10. Rendering or annotation failure cannot change the checker or persistence result.

Phase 0: Deterministic Characterization

Add synthetic OHLCV fixtures based on piker.data._source.def_iohlcv_fields; avoid brokers, networking, real actor trees, and Qt in the core suite.

Analysis tests

Create tests/test_tsp_analysis.py covering:

• uniform 1-second and 60-second series;
• one and multiple positive gaps;
• duplicate plus out-of-order timestamps;
• stale derived columns after sort/dedupe;
• zero, one, and multiple null segments;
• nulls at frame boundaries and nonzero absolute indexes;
• NumPy and Polars inputs;
• smart-dedupe conflict cases, including differing closes.

Backfill tests

Create tests/test_history_backfill.py with scripted get_hist, fake SHM, fake sampler stream, and recording storage implementations. Characterize:

• DataUnavailable completion without a hang;
• a genuinely empty frame;
• exact adjacency, overlap, and a positive storage gap;
• inclusive provider seam duplication;
• one-row and two-row startup history;
• cancellation after query, after SHM mutation, and before persistence;
• one and multiple null rows with boundary preservation.

Storage tests

Create tests/test_storage_nativedb.py using tmp_path:

• NumPy and Polars round trips;
• preservation of history older than the SHM window;
• duplicate conflict policy;
• write failure leaves the prior parquet readable;
• cache changes only after a successful commit;
• separate FQME/timeframe series do not interfere.

Annotation tests

Create tests/test_gap_annotations.py around a fake AnnotCtl, without Qt initially:

• left-close/right-open rectangle endpoints;
• up, down, and flat price direction;
• first-row and missing-prior-row behavior;
• empty gaps and out-of-range timestamps;
• deterministic batched rect/arrow/text payloads.

Gate: every confirmed defect above has a focused failing test or a written reason it requires a later integration test.

Phase 1: Canonical Frame and Gap Model

Refactor piker/tsp/_anal.py around one normalization order:

1. validate required columns and finite timestamps;
2. sort by time;
3. resolve duplicate timestamps under an explicit policy;
4. recompute all datetime/delta columns with with_dts();
5. reject/report non-positive ordering deltas;
6. detect positive sample-step gaps.

Keep the first implementation small: a normalized Polars frame may remain the contract, but each gap row must carry explicit time_prev, index_prev, close_prev, current time/index/open, duration, and missing-sample count.

Rewrite null grouping from extracted index/time arrays so the NumPy and Polars paths share one implementation. Treat a single null row as a normal one-row segment and honor a precomputed null_segs argument without rescanning.

Gate: analysis tests pass and no deduped/sorted frame retains stale delta columns.

Phase 2: Structured Backfill Completion

Refactor start_backfill() and tsdb_backfill() before changing persistence semantics:

• remove completion events that are awaited only after the owning nursery has already joined its children, or make completion a guaranteed finally outcome;
• represent completed, provider_exhausted, cancelled, and failed as explicit results;
• check empty arrays before indexing endpoints;
• normalize each provider frame and recompute endpoint metadata from the normalized data;
• define and enforce a disjoint seam between each new reverse frame and the already-published frame;
• update cursor state immediately with the SHM mutation, before any optional notification checkpoint;
• serialize reverse backfill and null repair unless a backend capability explicitly permits concurrent history requests;
• enforce declared backend request/rate limits.

Prefer structured task return values over actor-local events whose only consumer is the parent task.

Gate: exhaustion and cancellation tests terminate within their timeout, and SHM timestamps remain strictly increasing across all frame seams.

Phase 3: Non-Destructive Atomic NativeDB Writes

Stop treating the bounded SHM view as the full database.

Add a NativeDB merge/update operation that:

• accepts only the normalized provider frame being committed;
• loads or uses the cached persisted frame;
• merges by timestamp while preserving unaffected rows;
• uses the historical provider bar as authoritative at a closed-bar seam;
• validates ordering, uniqueness, schema, and nonzero timestamps before writing;
• writes to a same-directory temporary parquet;
• reopens and validates the temporary file;
• atomically replaces the final path;
• updates _dfs only after replacement succeeds;
• serializes writers per (fqme, timeframe).

Backfill commits the validated frame/delta, never a live shm.array snapshot containing mutable current bars or intentional zero space.

Gate: a persisted series can only grow backward or replace explicitly overlapping timestamps; its earliest timestamp never moves forward during ordinary backfill.

Phase 4: Exact Null Repair

Use the canonical null spans from Phase 1:

• query using explicit boundary timestamps;
• trim provider responses to the actual missing interval;
• write only null rows, never either valid boundary row;
• validate repaired timestamps and OHLC fields before publication or persistence;
• leave expected venue closures sparse;
• do not persist synthetic flat bars without a durable quality marker and an explicit schema decision.

If provider data is unavailable, retain and report the missing interval instead of silently converting it into apparently valid history.

Gate: single and multi-row repairs preserve both boundary bars and either eliminate exactly the target nulls or return an unresolved-gap result.

Phase 5: Venue and Provider Classification

Apply gap classification after canonical detection rather than embedding it in individual warning paths.

For IB:

• correct expiry comparison and weekday constants;
• classify internal positive gaps as well as trailing request-coverage gaps;
• require closure interval alignment, not merely the presence of a weekend/holiday somewhere in the gap;
• cover overnight sessions, weekends, holidays, DST, active/expired futures, and unknown calendars;
• replace production breakpoint() calls with structured data-quality results/errors.

Provider adapters must return normalized endpoint metadata and translate their native inclusive/exclusive APIs into the middleware boundary contract.

Gate: every gap is classified as expected closure, suspicious missing data, provider exhaustion, or invalid ordering; no production path pauses interactively.

Phase 6: Annotation Correctness and Performance

Extract a synchronous, Qt-free spec builder from markup_gaps():

build_gap_annotation_specs(frame, gaps)

It consumes the explicit gap endpoints from Phase 1 and:

• extracts needed columns once;
• replaces per-gap wdts.filter() with a vectorized join or indexed lookup;
• handles equal prices explicitly;
• preserves previous-close/current-open semantics;
• styles expected closures separately from suspicious gaps;
• emits deterministic rect/arrow/text batches.

Then fix remote lookup/reposition bounds by masking searchsorted() results before indexing and enforce both FQME and timeframe ownership during redraw.

Benchmark the same 1,285-gap workload from the Claude handoff. Treat these as local benchmark goals, not portable CI limits:

• client spec build under 20 ms;
• server preparation under 10 ms;
• one composite gap graphics item;
• no full-frame Polars filter per gap;
• total local creation near or below 150 ms.

Gate: annotation contract tests pass before performance numbers are accepted.

Phase 7: Crash and Provider Qualification

Run deterministic tests first, then the historical manual matrix from issue #62 on disposable storage:

• fresh full backfill;
• append to existing history;
• graceful Ctrl-C;
• network interruption;
• SIGTERM and SIGKILL during query and parquet commit;
• restart and revalidate after every interruption.

Start with Binance/Kraken/Kucoin, then run IB only with an available account and gateway. Record before/after earliest timestamp, latest timestamp, row count, unique count, null count, non-positive deltas, and parquet readability.

Verification Commands

Use an outer process timeout because actor teardown has a known second-runtime wedge:

timeout -k 5 30 python -m pytest -q \
  tests/test_tsp_analysis.py \
  tests/test_storage_nativedb.py

timeout -k 5 60 python -m pytest -q \
  tests/test_history_backfill.py

timeout -k 5 60 python -m pytest -q \
  tests/test_gap_annotations.py

CI=1 timeout -k 20 120 python -m pytest -q -rs \
  tests/test_feeds.py

Run live/network tests separately from deterministic tests. Do not automate piker store anal or piker store ldshm: both may pause interactively or mutate attached data.

Commit Boundaries

Keep implementation reviewable in this order:

1. synthetic fixtures and analysis regressions;
2. canonical normalization and null grouping;
3. backfill completion and frame-boundary fixes;
4. NativeDB merge and atomic persistence;
5. exact null repair;
6. IB/provider gap classification;
7. annotation contract fixes and vectorization;
8. qualification notes and documentation.

Each boundary must leave deterministic tests green. Do not mix renderer optimization into storage/backfill commits.

Deferred

The following remain follow-ups until the correctness gates above pass:

• chart context-menu/manual piecewise refill UX;
• automatic checker integration into every live chart;
• compact columnar annotation IPC;
• persistent synthetic-bar quality metadata/schema;
• full Deribit history repair and broad provider cleanup;
• Marketstore behavior changes;
• long-term incremental parquet partitioning.

Source References

• gitea/62.md: workaround-era backfill and manual test history
• gitea/71.md: IB venue closure behavior and pending account verification
• gitea/75.md: gap annotation, storage races, and backfiller follow-ups
• notes_to_self/claudy/gapsannotator_sesh.md: current annotation benchmark and client bottleneck
• https://github.com/pikers/piker/pull/486
• https://github.com/pikers/piker/pull/446
• https://github.com/pikers/piker/issues/536