piker/plans/opencode/gap-checker-backfiller-repair.md

# Gap Checker and Backfiller Repair

## Purpose

Continue the time-series gap checker and backfiller work
from PRs/issues `#62`, `#71`, and `#75`, prioritizing data
correctness and crash safety before annotation UX or
rendering performance.

Source baseline:

- branch: `flake_update`
- commit: `c6ec3d41`
- gap handoff:
  `notes_to_self/claudy/gapsannotator_sesh.md`
- primary code:
  `piker/tsp/_anal.py`, `piker/tsp/_history.py`,
  `piker/storage/nativedb.py`, and
  `piker/tsp/_annotate.py`

## Goals

1. Make gap detection operate on sorted, unique,
   validated OHLCV frames with fresh derived columns.
2. Make backfill completion, frame boundaries, and null
   repair deterministic under exhaustion and cancellation.
3. Prevent a bounded or transient SHM view from
   destructively replacing older persisted history.
4. Persist normalized history through crash-safe atomic
   file replacement.
5. Distinguish expected venue closures from suspicious
   missing data without production breakpoints.
6. Generate correct annotation specs before optimizing
   the remaining client-side Polars bottleneck.

## Confirmed Baseline Defects

These are current-code findings to characterize in tests,
not assumptions inferred from historical task state.

### Gap normalization

- `dedupe()` computes `time_prev`, `s_diff`, and
  `dt_diff` before sorting/deduplication, then returns
  stale derived columns in the normalized frame
  (`piker/tsp/_anal.py:683-729`).
- Negative/out-of-order deltas are treated as ordinary
  gaps because `detect_time_gaps()` filters on
  `abs(s_diff)` (`piker/tsp/_anal.py:582-638`).
- `get_null_segs()` drops single zero rows and its Polars
  path does not define all state used by the shared path
  (`piker/tsp/_anal.py:227-471`).
- `iter_null_segs()` ignores a supplied `null_segs`
  result and rescans the frame (`piker/tsp/_anal.py:474-546`).

### Backfill control and boundaries

- `DataUnavailable` returns before `bf_done.set()`, while
  `tsdb_backfill()` later waits forever on that event
  (`piker/tsp/_history.py:516-536,1193-1212,1302-1308`).
- Empty arrays are indexed before the nominal null-frame
  branch can handle them (`piker/tsp/_history.py:538-601`).
- Provider frame seams have no explicit inclusive/exclusive
  contract, allowing duplicate adjacent timestamps in SHM.
- Cancellation may occur after SHM mutation but before
  cursor and persistence bookkeeping
  (`piker/tsp/_history.py:661-789`).

### Null repair

- Null query length is derived from an absolute endpoint,
  not the null-span width (`piker/tsp/_history.py:278-287`).
- Fallback repair includes and mutates the valid leading
  boundary row (`piker/tsp/_history.py:345-367`).
- Null repair and reverse backfill share one provider
  client concurrently without enforcing the backend's
  declared request limits.

### Persistence

- Every iteration passes bounded `shm.array` to
  `write_ohlcv()`, while NativeDB replaces the complete
  parquet file. Older rows that do not fit in SHM can be
  lost (`piker/tsp/_history.py:722-789`,
  `piker/storage/nativedb.py:305-368`).
- NativeDB writes directly to the final path and updates
  its cache before the write succeeds; interruption can
  leave disk and cache partial or inconsistent.
- Intentional zero-filled SHM space may be persisted while
  null repair and reverse backfill run concurrently.

### Annotation contract

- `markup_gaps()` performs a full Polars filter per gap and
  remains the measured client bottleneck
  (`piker/tsp/_annotate.py:139-217`).
- Its missing-prior-row fallback selects the current row,
  and `copysign(1, 0)` makes the flat-price branch
  unreachable (`piker/tsp/_annotate.py:166-239`).
- Remote/reposition timestamp lookups index
  `searchsorted()` results before excluding values equal
  to `len(array)`.

## Design Invariants

All implementation phases preserve these rules:

1. Persisted provider rows have nonzero, finite timestamps.
2. Normalized timestamps are strictly increasing and
   unique.
3. Time-derived columns are computed only after final
   sorting and duplicate resolution.
4. A detected gap explicitly carries its left and right
   endpoints; consumers do not infer the left row from
   `index - 1`.
5. Provider frame metadata equals the normalized array's
   actual first and last timestamps.
6. Internal middleware uses one documented boundary
   convention; provider adapters translate into it.
7. Expected sparse venue closures are represented by
   timestamp distance, not persisted zero or synthetic
   bars.
8. Storage merges preserve all unaffected older rows.
9. A storage commit becomes visible atomically, and cache
   state changes only after disk commit succeeds.
10. Rendering or annotation failure cannot change the
    checker or persistence result.

## Phase 0: Deterministic Characterization

Add synthetic OHLCV fixtures based on
`piker.data._source.def_iohlcv_fields`; avoid brokers,
networking, real actor trees, and Qt in the core suite.

### Analysis tests

Create `tests/test_tsp_analysis.py` covering:

- uniform 1-second and 60-second series;
- one and multiple positive gaps;
- duplicate plus out-of-order timestamps;
- stale derived columns after sort/dedupe;
- zero, one, and multiple null segments;
- nulls at frame boundaries and nonzero absolute indexes;
- NumPy and Polars inputs;
- smart-dedupe conflict cases, including differing closes.

### Backfill tests

Create `tests/test_history_backfill.py` with scripted
`get_hist`, fake SHM, fake sampler stream, and recording
storage implementations. Characterize:

- `DataUnavailable` completion without a hang;
- a genuinely empty frame;
- exact adjacency, overlap, and a positive storage gap;
- inclusive provider seam duplication;
- one-row and two-row startup history;
- cancellation after query, after SHM mutation, and
  before persistence;
- one and multiple null rows with boundary preservation.

### Storage tests

Create `tests/test_storage_nativedb.py` using `tmp_path`:

- NumPy and Polars round trips;
- preservation of history older than the SHM window;
- duplicate conflict policy;
- write failure leaves the prior parquet readable;
- cache changes only after a successful commit;
- separate FQME/timeframe series do not interfere.

### Annotation tests

Create `tests/test_gap_annotations.py` around a fake
`AnnotCtl`, without Qt initially:

- left-close/right-open rectangle endpoints;
- up, down, and flat price direction;
- first-row and missing-prior-row behavior;
- empty gaps and out-of-range timestamps;
- deterministic batched rect/arrow/text payloads.

Gate: every confirmed defect above has a focused failing
test or a written reason it requires a later integration
test.

## Phase 1: Canonical Frame and Gap Model

Refactor `piker/tsp/_anal.py` around one normalization
order:

1. validate required columns and finite timestamps;
2. sort by `time`;
3. resolve duplicate timestamps under an explicit policy;
4. recompute all datetime/delta columns with `with_dts()`;
5. reject/report non-positive ordering deltas;
6. detect positive sample-step gaps.

Keep the first implementation small: a normalized Polars
frame may remain the contract, but each gap row must carry
explicit `time_prev`, `index_prev`, `close_prev`, current
`time/index/open`, duration, and missing-sample count.

Rewrite null grouping from extracted index/time arrays so
the NumPy and Polars paths share one implementation. Treat
a single null row as a normal one-row segment and honor a
precomputed `null_segs` argument without rescanning.

Gate: analysis tests pass and no deduped/sorted frame
retains stale delta columns.

## Phase 2: Structured Backfill Completion

Refactor `start_backfill()` and `tsdb_backfill()` before
changing persistence semantics:

- remove completion events that are awaited only after the
  owning nursery has already joined its children, or make
  completion a guaranteed `finally` outcome;
- represent `completed`, `provider_exhausted`,
  `cancelled`, and `failed` as explicit results;
- check empty arrays before indexing endpoints;
- normalize each provider frame and recompute endpoint
  metadata from the normalized data;
- define and enforce a disjoint seam between each new
  reverse frame and the already-published frame;
- update cursor state immediately with the SHM mutation,
  before any optional notification checkpoint;
- serialize reverse backfill and null repair unless a
  backend capability explicitly permits concurrent
  history requests;
- enforce declared backend request/rate limits.

Prefer structured task return values over actor-local
events whose only consumer is the parent task.

Gate: exhaustion and cancellation tests terminate within
their timeout, and SHM timestamps remain strictly
increasing across all frame seams.

## Phase 3: Non-Destructive Atomic NativeDB Writes

Stop treating the bounded SHM view as the full database.

Add a NativeDB merge/update operation that:

- accepts only the normalized provider frame being
  committed;
- loads or uses the cached persisted frame;
- merges by timestamp while preserving unaffected rows;
- uses the historical provider bar as authoritative at a
  closed-bar seam;
- validates ordering, uniqueness, schema, and nonzero
  timestamps before writing;
- writes to a same-directory temporary parquet;
- reopens and validates the temporary file;
- atomically replaces the final path;
- updates `_dfs` only after replacement succeeds;
- serializes writers per `(fqme, timeframe)`.

Backfill commits the validated frame/delta, never a live
`shm.array` snapshot containing mutable current bars or
intentional zero space.

Gate: a persisted series can only grow backward or replace
explicitly overlapping timestamps; its earliest timestamp
never moves forward during ordinary backfill.

## Phase 4: Exact Null Repair

Use the canonical null spans from Phase 1:

- query using explicit boundary timestamps;
- trim provider responses to the actual missing interval;
- write only null rows, never either valid boundary row;
- validate repaired timestamps and OHLC fields before
  publication or persistence;
- leave expected venue closures sparse;
- do not persist synthetic flat bars without a durable
  quality marker and an explicit schema decision.

If provider data is unavailable, retain and report the
missing interval instead of silently converting it into
apparently valid history.

Gate: single and multi-row repairs preserve both boundary
bars and either eliminate exactly the target nulls or
return an unresolved-gap result.

## Phase 5: Venue and Provider Classification

Apply gap classification after canonical detection rather
than embedding it in individual warning paths.

For IB:

- correct expiry comparison and weekday constants;
- classify internal positive gaps as well as trailing
  request-coverage gaps;
- require closure interval alignment, not merely the
  presence of a weekend/holiday somewhere in the gap;
- cover overnight sessions, weekends, holidays, DST,
  active/expired futures, and unknown calendars;
- replace production `breakpoint()` calls with structured
  data-quality results/errors.

Provider adapters must return normalized endpoint metadata
and translate their native inclusive/exclusive APIs into
the middleware boundary contract.

Gate: every gap is classified as expected closure,
suspicious missing data, provider exhaustion, or invalid
ordering; no production path pauses interactively.

## Phase 6: Annotation Correctness and Performance

Extract a synchronous, Qt-free spec builder from
`markup_gaps()`:

```python
build_gap_annotation_specs(frame, gaps)
```

It consumes the explicit gap endpoints from Phase 1 and:

- extracts needed columns once;
- replaces per-gap `wdts.filter()` with a vectorized join
  or indexed lookup;
- handles equal prices explicitly;
- preserves previous-close/current-open semantics;
- styles expected closures separately from suspicious
  gaps;
- emits deterministic rect/arrow/text batches.

Then fix remote lookup/reposition bounds by masking
`searchsorted()` results before indexing and enforce both
FQME and timeframe ownership during redraw.

Benchmark the same 1,285-gap workload from the Claude
handoff. Treat these as local benchmark goals, not portable
CI limits:

- client spec build under 20 ms;
- server preparation under 10 ms;
- one composite gap graphics item;
- no full-frame Polars filter per gap;
- total local creation near or below 150 ms.

Gate: annotation contract tests pass before performance
numbers are accepted.

## Phase 7: Crash and Provider Qualification

Run deterministic tests first, then the historical manual
matrix from issue `#62` on disposable storage:

- fresh full backfill;
- append to existing history;
- graceful Ctrl-C;
- network interruption;
- SIGTERM and SIGKILL during query and parquet commit;
- restart and revalidate after every interruption.

Start with Binance/Kraken/Kucoin, then run IB only with an
available account and gateway. Record before/after earliest
timestamp, latest timestamp, row count, unique count, null
count, non-positive deltas, and parquet readability.

## Verification Commands

Use an outer process timeout because actor teardown has a
known second-runtime wedge:

```bash
timeout -k 5 30 python -m pytest -q \
  tests/test_tsp_analysis.py \
  tests/test_storage_nativedb.py

timeout -k 5 60 python -m pytest -q \
  tests/test_history_backfill.py

timeout -k 5 60 python -m pytest -q \
  tests/test_gap_annotations.py

CI=1 timeout -k 20 120 python -m pytest -q -rs \
  tests/test_feeds.py
```

Run live/network tests separately from deterministic tests.
Do not automate `piker store anal` or `piker store ldshm`:
both may pause interactively or mutate attached data.

## Commit Boundaries

Keep implementation reviewable in this order:

1. synthetic fixtures and analysis regressions;
2. canonical normalization and null grouping;
3. backfill completion and frame-boundary fixes;
4. NativeDB merge and atomic persistence;
5. exact null repair;
6. IB/provider gap classification;
7. annotation contract fixes and vectorization;
8. qualification notes and documentation.

Each boundary must leave deterministic tests green. Do not
mix renderer optimization into storage/backfill commits.

## Deferred

The following remain follow-ups until the correctness
gates above pass:

- chart context-menu/manual piecewise refill UX;
- automatic checker integration into every live chart;
- compact columnar annotation IPC;
- persistent synthetic-bar quality metadata/schema;
- full Deribit history repair and broad provider cleanup;
- Marketstore behavior changes;
- long-term incremental parquet partitioning.

## Source References

- `gitea/62.md`: workaround-era backfill and manual test
  history
- `gitea/71.md`: IB venue closure behavior and pending
  account verification
- `gitea/75.md`: gap annotation, storage races, and
  backfiller follow-ups
- `notes_to_self/claudy/gapsannotator_sesh.md`: current
  annotation benchmark and client bottleneck
- https://github.com/pikers/piker/pull/486
- https://github.com/pikers/piker/pull/446
- https://github.com/pikers/piker/issues/536