piker/docs/manual_backfill_qualification.rst

Manual Backfill Qualification

Purpose

This runbook separates provider coverage, durable NativeDB state, shared-memory hydration, and chart rendering while reproducing gappy history. Work one FQME and timeframe at a time. Preserve the original Parquet first, then qualify both repair of that evidence and a completely fresh backfill.

The read-only command used throughout is:

piker store audit FQME --period 60
piker store audit FQME --period 60 --json

It reads raw Parquet directly. It does not open a storage client, canonicalize data, publish cache state, attach SHM, or rewrite history. Positive gaps are deliberately unclassified; an observed interval is not automatically corruption or an expected venue closure.

Warning

Do not run qualification against normal user storage. The helper below requires a marked disposable root before it will seed, archive, or clear a Parquet. piker store anal and piker store ldshm are interactive and potentially mutating; they are not audit commands.

Keep the disposable root private and do not rename, replace, or symlink files under it while a helper is running. Stop the chart cleanly before every audit or comparison phase.

Load The Xonsh Helpers

Run from the repository root in the same xonsh which will launch the chart or daemon:

source snippets/nativedb_backfill_audit.xsh

Choose a disposable root and copy only the provider configuration needed for the case:

run_root = p'/tmp/piker-backfill/mnq-20260918-60s'
source_config = p'~/.config/piker'.expanduser()
bfq_init(run_root, source_config)
bfq_record_case(
    run_root,
    fqme='mnq.cme.20260918.ib',
    period_s=60,
    provider='ib',
    symptom='gappy chart during reverse backfill',
    bad_start_utc=None,
    bad_end_utc=None,
)

bfq_init() sets XDG_CONFIG_HOME in the current xonsh, so every subsequent piker and pikerd child inherits the disposable config. The effective storage path is under RUN_ROOT/xdg/piker/nativedb. It also records the exact repository root and piker executable in the qualification marker, prepends that root to child PYTHONPATH, and verifies that the bound executable exposes piker store audit. Audits fail if that executable imports another tree.

The copied brokers.toml can contain credentials. Keep the run root private and remove it manually after evidence is no longer needed.

Repository Preflight

Before every daemon start, verify the code and config inherited by child processes:

git status --short --branch
git rev-parse HEAD
python -c 'import piker; print(piker.__file__)'
python -c 'import os; print(os.environ["XDG_CONFIG_HOME"])'

Stop any daemon from the previous phase cleanly. Use a separate registry address in the disposable conf.toml when a normal Piker stack remains active.

Known-Bad Replay

Copy the original file; never move it out of normal storage:

fqme = 'mnq.cme.20260918.ib'
period_s = 60
source_parquet = (
    p'~/.config/piker/nativedb'.expanduser()
    /f'{fqme}.ohlcv{period_s}s.parquet'
)
bfq_seed(run_root, fqme, period_s, source_parquet)
bfq_audit(run_root, 'baseline', fqme, period_s)

The baseline phase writes:

evidence/baseline.json
evidence/baseline.parquet
evidence/baseline.manual.json

The JSON checksum must match the read-only snapshot, and the manual sidecar binds that report to its phase and case. Comparison rechecks those bindings before reading either phase. The report records raw schema, row and timestamp counts, non-finite values, ordering, duplicate excess, canonical index state, and every retained positive-gap detail. Files with invalid Parquet encoding can not produce a JSON report, but --snapshot still preserves their exact bytes before parse failure.

Launch the chart manually from the same shell and capture logs:

piker -l info chart mnq.cme.20260918.ib \
    2>&1 | tee @(run_root / 'logs' / 'repair-chart.log')

After a clean stop:

bfq_audit(
    run_root,
    'repair',
    fqme,
    period_s,
    require_structural=True,
)
bfq_compare(run_root, 'baseline', 'repair')

The comparison fails if structural validity regresses, any original timestamp disappears, chronological endpoints shrink, timestamps become duplicated or unordered, indexes cease to be canonical, or OHLCV becomes non-finite. Provider replacement of matching seam rows is reported separately as changed_common_timestamps for operator review.

Fresh Backfill

Yes, fresh qualification means saving the known-bad evidence and then starting with no active Parquet in the disposable NativeDB. Archive only the disposable file:

bfq_clear_for_fresh(run_root, fqme, period_s)

Confirm the printed active path is absent, launch the same chart, wait for bounded provider exhaustion or the intended coverage, then stop cleanly:

bfq_audit(
    run_root,
    'fresh',
    fqme,
    period_s,
    require_structural=True,
)

Do not compare baseline to fresh as a timestamp-preservation gate: provider limits may intentionally produce a different initial extent. Treat fresh as the baseline for restart and append qualification.

Restart And Append

Restart the same chart against the fresh persisted file, stop cleanly, and record:

bfq_audit(
    run_root,
    'restart',
    fqme,
    period_s,
    require_structural=True,
)
bfq_compare(run_root, 'fresh', 'restart')

Leave a subsequent run active long enough to append new samples, stop, and record:

bfq_audit(
    run_root,
    'append',
    fqme,
    period_s,
    require_structural=True,
)
bfq_compare(
    run_root,
    'restart',
    'append',
    require_newer=True,
)

The comparison JSON reports gaps added and removed. Expected closures stay visible until a venue-aware classifier proves their session alignment.

Gap Layer Triage

Use the persisted report and chart together:

Observation	Suspected layer
Parquet contiguous, chart gappy	SHM hydration, sampling, or chart render
Parquet gap, provider has bars	provider-delta or storage merge
Provider and Parquet both gappy	provider omission or venue closure
Gap vanishes after restart	SHM publication or cache lifecycle
Old timestamps disappear	destructive persistence regression
Newest seam duplicates	conflict or dedupe policy

Always record exact UTC gap endpoints. The audit's numeric epoch is authoritative; the *_utc fields are deterministic display values.

Expansion Matrix

Expand only after the fixed FQME passes fresh, restart, and append. Use a new disposable root and case record for every row:

Case	Period	Coverage model
Original IB future	60 seconds	CME maintenance and weekend closures
Original IB future	1 second	high-frequency provider limits
Second CME future	60 seconds	contract-specific history
IB equity	60 seconds	regular overnight and weekend closure
IB FX	60 seconds	24x5 session behavior
Crypto spot/perp	60 seconds	continuous-session control
Expired future	60 seconds	history-only and expiry behavior

A sourceable matrix can remain ordinary Python data in xonsh:

cases = [
    {
        'name': 'mnq-60s',
        'fqme': 'mnq.cme.20260918.ib',
        'period_s': 60,
        'provider': 'ib',
    },
    {
        'name': 'mnq-1s',
        'fqme': 'mnq.cme.20260918.ib',
        'period_s': 1,
        'provider': 'ib',
    },
]
for case in cases:
    case_root = p'/tmp/piker-backfill' / case['name']
    bfq_init(case_root, source_config)
    bfq_record_case(
        case_root,
        fqme=case['fqme'],
        period_s=case['period_s'],
        provider=case['provider'],
        symptom='manual expansion case',
    )

This loop prepares isolated case roots only. Run the complete seed or fresh, chart, audit, restart, append, and compare lifecycle in each root; directory creation alone is not qualification.

The chart path may allocate companion 1-second and 60-second files. Audit the selected case path explicitly and retain companion files as secondary evidence; do not imply that the live request isolated one timeframe.

Interruption Qualification

After normal restart is stable, interrupt one phase with graceful Ctrl-C, restart, and compare against the last clean evidence. Provider disconnect, SIGTERM, and deterministic write-barrier failures come next. Avoid timing a SIGKILL by hand; add an explicit failpoint before qualifying atomic replace boundaries.

Result Interpretation

structural_ok means the raw persisted file satisfies durable schema, finite values, positive unique ordered timestamps, canonical index, and period-aligned gaps. gap_free means no positive gaps were observed when at least two valid timestamps made cadence verifiable; otherwise it is null. qualification_ok combines both and is intentionally strict.

A structurally valid session-market file can have gap_free = false. Those intervals require provider/session evidence before being marked as expected closures. Unknown calendar state must stay unknown.