Manual Backfill Qualification ============================= Purpose ------- This runbook separates provider coverage, durable NativeDB state, shared-memory hydration, and chart rendering while reproducing gappy history. Work one FQME and timeframe at a time. Preserve the original Parquet first, then qualify both repair of that evidence and a completely fresh backfill. The read-only command used throughout is:: piker store audit FQME --period 60 piker store audit FQME --period 60 --json It reads raw Parquet directly. It does not open a storage client, canonicalize data, publish cache state, attach SHM, or rewrite history. Positive gaps are deliberately unclassified; an observed interval is not automatically corruption or an expected venue closure. .. warning:: Do not run qualification against normal user storage. The helper below requires a marked disposable root before it will seed, archive, or clear a Parquet. ``piker store anal`` and ``piker store ldshm`` are interactive and potentially mutating; they are not audit commands. Keep the disposable root private and do not rename, replace, or symlink files under it while a helper is running. Stop the chart cleanly before every audit or comparison phase. Load The Xonsh Helpers ---------------------- Run from the repository root in the same xonsh which will launch the chart or daemon:: source snippets/nativedb_backfill_audit.xsh Choose a disposable root and copy only the provider configuration needed for the case:: run_root = p'/tmp/piker-backfill/mnq-20260918-60s' source_config = p'~/.config/piker'.expanduser() bfq_init(run_root, source_config) bfq_record_case( run_root, fqme='mnq.cme.20260918.ib', period_s=60, provider='ib', symptom='gappy chart during reverse backfill', bad_start_utc=None, bad_end_utc=None, ) ``bfq_init()`` sets ``XDG_CONFIG_HOME`` in the current xonsh, so every subsequent ``piker`` and ``pikerd`` child inherits the disposable config. The effective storage path is under ``RUN_ROOT/xdg/piker/nativedb``. It also records the exact repository root and ``piker`` executable in the qualification marker, prepends that root to child ``PYTHONPATH``, and verifies that the bound executable exposes ``piker store audit``. Audits fail if that executable imports another tree. The copied ``brokers.toml`` can contain credentials. Keep the run root private and remove it manually after evidence is no longer needed. Repository Preflight -------------------- Before every daemon start, verify the code and config inherited by child processes:: git status --short --branch git rev-parse HEAD python -c 'import piker; print(piker.__file__)' python -c 'import os; print(os.environ["XDG_CONFIG_HOME"])' Stop any daemon from the previous phase cleanly. Use a separate registry address in the disposable ``conf.toml`` when a normal Piker stack remains active. Known-Bad Replay ---------------- Copy the original file; never move it out of normal storage:: fqme = 'mnq.cme.20260918.ib' period_s = 60 source_parquet = ( p'~/.config/piker/nativedb'.expanduser() /f'{fqme}.ohlcv{period_s}s.parquet' ) bfq_seed(run_root, fqme, period_s, source_parquet) bfq_audit(run_root, 'baseline', fqme, period_s) The baseline phase writes:: evidence/baseline.json evidence/baseline.parquet evidence/baseline.manual.json The JSON checksum must match the read-only snapshot, and the manual sidecar binds that report to its phase and case. Comparison rechecks those bindings before reading either phase. The report records raw schema, row and timestamp counts, non-finite values, ordering, duplicate excess, canonical index state, and every retained positive-gap detail. Files with invalid Parquet encoding can not produce a JSON report, but ``--snapshot`` still preserves their exact bytes before parse failure. Launch the chart manually from the same shell and capture logs:: piker -l info chart mnq.cme.20260918.ib \ 2>&1 | tee @(run_root / 'logs' / 'repair-chart.log') After a clean stop:: bfq_audit( run_root, 'repair', fqme, period_s, require_structural=True, ) bfq_compare(run_root, 'baseline', 'repair') The comparison fails if structural validity regresses, any original timestamp disappears, chronological endpoints shrink, timestamps become duplicated or unordered, indexes cease to be canonical, or OHLCV becomes non-finite. Provider replacement of matching seam rows is reported separately as ``changed_common_timestamps`` for operator review. Fresh Backfill -------------- Yes, fresh qualification means saving the known-bad evidence and then starting with no active Parquet in the disposable NativeDB. Archive only the disposable file:: bfq_clear_for_fresh(run_root, fqme, period_s) Confirm the printed active path is absent, launch the same chart, wait for bounded provider exhaustion or the intended coverage, then stop cleanly:: bfq_audit( run_root, 'fresh', fqme, period_s, require_structural=True, ) Do not compare ``baseline`` to ``fresh`` as a timestamp-preservation gate: provider limits may intentionally produce a different initial extent. Treat ``fresh`` as the baseline for restart and append qualification. Restart And Append ------------------ Restart the same chart against the fresh persisted file, stop cleanly, and record:: bfq_audit( run_root, 'restart', fqme, period_s, require_structural=True, ) bfq_compare(run_root, 'fresh', 'restart') Leave a subsequent run active long enough to append new samples, stop, and record:: bfq_audit( run_root, 'append', fqme, period_s, require_structural=True, ) bfq_compare( run_root, 'restart', 'append', require_newer=True, ) The comparison JSON reports gaps added and removed. Expected closures stay visible until a venue-aware classifier proves their session alignment. Gap Layer Triage ---------------- Use the persisted report and chart together: =============================== ========================================= Observation Suspected layer =============================== ========================================= Parquet contiguous, chart gappy SHM hydration, sampling, or chart render Parquet gap, provider has bars provider-delta or storage merge Provider and Parquet both gappy provider omission or venue closure Gap vanishes after restart SHM publication or cache lifecycle Old timestamps disappear destructive persistence regression Newest seam duplicates conflict or dedupe policy =============================== ========================================= Always record exact UTC gap endpoints. The audit's numeric epoch is authoritative; the ``*_utc`` fields are deterministic display values. Expansion Matrix ---------------- Expand only after the fixed FQME passes fresh, restart, and append. Use a new disposable root and case record for every row: ==================== ========== ======================================== Case Period Coverage model ==================== ========== ======================================== Original IB future 60 seconds CME maintenance and weekend closures Original IB future 1 second high-frequency provider limits Second CME future 60 seconds contract-specific history IB equity 60 seconds regular overnight and weekend closure IB FX 60 seconds 24x5 session behavior Crypto spot/perp 60 seconds continuous-session control Expired future 60 seconds history-only and expiry behavior ==================== ========== ======================================== A sourceable matrix can remain ordinary Python data in xonsh:: cases = [ { 'name': 'mnq-60s', 'fqme': 'mnq.cme.20260918.ib', 'period_s': 60, 'provider': 'ib', }, { 'name': 'mnq-1s', 'fqme': 'mnq.cme.20260918.ib', 'period_s': 1, 'provider': 'ib', }, ] for case in cases: case_root = p'/tmp/piker-backfill' / case['name'] bfq_init(case_root, source_config) bfq_record_case( case_root, fqme=case['fqme'], period_s=case['period_s'], provider=case['provider'], symptom='manual expansion case', ) This loop prepares isolated case roots only. Run the complete seed or fresh, chart, audit, restart, append, and compare lifecycle in each root; directory creation alone is not qualification. The chart path may allocate companion 1-second and 60-second files. Audit the selected case path explicitly and retain companion files as secondary evidence; do not imply that the live request isolated one timeframe. Interruption Qualification -------------------------- After normal restart is stable, interrupt one phase with graceful Ctrl-C, restart, and compare against the last clean evidence. Provider disconnect, SIGTERM, and deterministic write-barrier failures come next. Avoid timing a SIGKILL by hand; add an explicit failpoint before qualifying atomic replace boundaries. Result Interpretation --------------------- ``structural_ok`` means the raw persisted file satisfies durable schema, finite values, positive unique ordered timestamps, canonical index, and period-aligned gaps. ``gap_free`` means no positive gaps were observed when at least two valid timestamps made cadence verifiable; otherwise it is ``null``. ``qualification_ok`` combines both and is intentionally strict. A structurally valid session-market file can have ``gap_free = false``. Those intervals require provider/session evidence before being marked as expected closures. Unknown calendar state must stay unknown.