piker/plans/opencode/known-fqme-backfill-qualification.md

# Known-FQME Backfill Qualification

## Purpose

Move `backfiller_deep_fixes` into the main checkout, reproduce the
historical backfill failures against one known-problem market, and turn
every confirmed failure into deterministic regression coverage before
expanding live qualification.

This is a correctness-debugging pass, not a broad provider smoke test.
One FQME and one timeframe remain fixed until their original failure is
understood, covered, repaired, and requalified.

## Current Repository State

The branch is currently checked out at:

`/home/goodboy/repos/piker/.claude/wkts/backfiller_deep_fixes`

Its head is `10e0ced7`. The linked worktree has only generated,
untracked commit-message metadata.

The main checkout is on `flake_update` at `c6ec3d41` and has staged and
untracked user work. Do not switch branches, stash, remove files, or
close the linked worktree until that state has been explicitly
preserved by the user.

## Phase 1: Transfer Branch Ownership Safely

1. Review the main checkout's staged and untracked state.
2. Preserve it using the user's chosen mechanism: finish and commit it,
   move it to another branch/worktree, or explicitly create a stash that
   includes the required untracked files.
3. Review the linked worktree's generated `.claude/skills/commit-msg/`
   artifacts. Archive or discard them explicitly; they are not branch
   source.
4. Remove the linked `backfiller_deep_fixes` worktree while keeping its
   exact branch.
5. Switch the main checkout to `backfiller_deep_fixes`.
6. Verify all entry points import from the main checkout and resolve to
   head `10e0ced7` or its descendant.
7. Run the 31-test deterministic analysis/backfill/storage set before
   booting any daemon.

Gate: the main checkout owns `backfiller_deep_fixes`, tracked status is
clean, user work from `flake_update` is preserved, and deterministic
tests pass.

## Phase 2: Record The Known Failure

Capture these details for the selected case:

- provider and exact FQME;
- timeframe, initially `60` or `1` seconds;
- known-bad date/time range and timezone;
- original symptom: destructive truncation, duplicate seam, null rows,
  out-of-order timestamps, hang, or another observed failure;
- command or chart workflow that exposed it;
- whether a known-bad parquet already exists;
- expected venue closure or continuous-session behavior;
- credentials, gateway, and network requirements;
- smallest reproducible provider response or persisted frame.

Do not start with multiple symbols, providers, or timeframes. Preserve a
checksum of any known-bad parquet before copying it into disposable
storage.

Gate: one case has an explicit expected result and enough source data to
distinguish a real provider/venue gap from a piker persistence defect.

## Phase 3: Build A Disposable Runtime

Use the main checkout's Python environment and a dedicated XDG config
root. The `pikerd` process and every client process must inherit the same
config root and import the main checkout.

The disposable directory contains only:

- a minimal `conf.toml`;
- credentials required for the selected provider;
- a dedicated `nativedb/` directory;
- an optional copy of the single known-bad parquet;
- captured logs and before/after audit output.

Ensure no older `pikerd` is listening on the selected registry address.
Verify `piker.__file__`, the branch, and the commit before starting the
daemon.

Gate: deleting the disposable directory cannot affect normal user
configuration, accounting data, or production history.

## Phase 4: Establish A Read-Only Baseline

Before backfill, record:

- parquet checksum and readability;
- earliest and latest timestamp;
- total and unique timestamp counts;
- duplicate count;
- zero and non-finite timestamp counts;
- non-positive timestamp deltas;
- null OHLCV counts;
- contiguous persisted index status;
- positive sample gaps with explicit endpoints.

Use a read-only audit helper or focused Python test. Do not automate
`piker store anal` or `piker store ldshm`; both can pause or mutate
attached data.

Gate: the original symptom is visible in saved evidence, or the plan
records why a fresh provider fetch is required to reproduce it.

## Phase 5: Reproduce One Lifecycle At A Time

Run in this order:

1. Load the known-bad persisted history without a provider write.
2. Start `pikerd` and request the chosen FQME/timeframe only.
3. Observe startup latest-frame query, pre-update TSDB snapshot, reverse
   backfill, provider-delta commit, SHM publication, and notification.
4. Stop cleanly and rerun the read-only audit.
5. Restart and append to the same disposable history.
6. Compare earliest/latest timestamps, row counts, conflicts, and gaps
   against the baseline.

Capture logs around every `update_ohlcv()` call, provider frame boundary,
SHM truncation, lock acquisition, temporary parquet, replacement, and
cache publication.

Gate: the run either preserves every unaffected row and normalizes the
known defect, or yields a bounded failure with enough data for a test.

## Phase 6: Convert Failures Into Tests

For each reproduced defect:

1. Save the smallest provider frames and persisted rows needed to
   reproduce it. Prefer synthetic arrays; use a trimmed fixture only
   when provider-specific boundaries matter.
2. Write a failing deterministic test before changing production code.
3. Assert exact timestamps, endpoint policy, persisted rows, and state
   transition ordering rather than only asserting completion.
4. Keep actor-free tests in the existing focused modules where possible.
5. Add subprocess integration tests only for process death, advisory
   locks, or atomic filesystem behavior that cannot be represented in
   one Trio runtime.
6. Apply the smallest production fix and rerun both the focused test and
   the complete deterministic set.
7. Rerun the exact manual FQME lifecycle that found the failure.

Potential deterministic modules:

- `tests/test_tsp_analysis.py` for normalization and classification;
- `tests/test_history_backfill.py` for provider/SHM/storage ordering;
- `tests/test_storage_nativedb.py` for merge and atomic durability;
- a new subprocess fault module for kill/restart and cross-process locks;
- provider-specific fixtures only when generic frames cannot reproduce
  the behavior.

Gate: no production fix lands without a failing-before, passing-after
test or a written explanation for an unavoidable live-only condition.

## Phase 7: Fault Qualification

After the normal lifecycle is stable, exercise:

1. graceful Ctrl-C during provider query and storage update;
2. SIGTERM at deterministic query/write barriers;
3. SIGKILL after temp write, before replace, and after replace;
4. provider disconnect and recovery;
5. concurrent NativeDB writers for the same series;
6. restart after every interruption;
7. actual sampler backpressure and actor teardown;
8. MarketStore separately from NativeDB.

Use explicit barriers or failpoints rather than timing sleeps whenever
the failure position matters. Repeat the read-only audit after every
restart.

Gate: the final parquet is always old-valid or new-valid, unaffected
history remains present, indexes are contiguous, cache agrees with the
visible file, and shutdown remains bounded.

## Expansion Order

Only after the chosen FQME passes:

1. qualify its second timeframe;
2. qualify another continuous-session provider;
3. run Binance, Kraken, and Kucoin cases;
4. test IB with an available account and gateway;
5. add expected venue-closure classification;
6. widen to the remaining manual matrix.

## Commit Boundaries

Keep follow-up commits narrow:

1. captured fixture and failing regression;
2. minimal analysis/backfill/storage correction;
3. subprocess fault-injection harness;
4. provider-specific adapter correction;
5. qualification notes and reproducible audit output.

Every boundary must leave deterministic tests green. Do not combine
annotation performance, unrelated providers, or chart UX with the known
FQME debugging loop.

## Exit Criteria

The selected FQME/timeframe is complete when:

- the original bug reproduces from preserved evidence;
- a deterministic regression covers it where technically possible;
- fresh and append backfills preserve unaffected history;
- provider seam conflicts follow the documented policy;
- no zero, duplicate, non-finite, or non-positive timestamps persist;
- indexes remain contiguous;
- clean shutdown and restart preserve readable parquet;
- failure injection leaves an old-valid or new-valid file;
- remaining live-only risks are explicitly recorded.