286 lines
9.9 KiB
ReStructuredText
286 lines
9.9 KiB
ReStructuredText
Manual Backfill Qualification
|
|
=============================
|
|
|
|
Purpose
|
|
-------
|
|
|
|
This runbook separates provider coverage, durable NativeDB state,
|
|
shared-memory hydration, and chart rendering while reproducing gappy
|
|
history. Work one FQME and timeframe at a time. Preserve the original
|
|
Parquet first, then qualify both repair of that evidence and a completely
|
|
fresh backfill.
|
|
|
|
The read-only command used throughout is::
|
|
|
|
piker store audit FQME --period 60
|
|
piker store audit FQME --period 60 --json
|
|
|
|
It reads raw Parquet directly. It does not open a storage client,
|
|
canonicalize data, publish cache state, attach SHM, or rewrite history.
|
|
Positive gaps are deliberately unclassified; an observed interval is not
|
|
automatically corruption or an expected venue closure.
|
|
|
|
.. warning::
|
|
|
|
Do not run qualification against normal user storage. The helper below
|
|
requires a marked disposable root before it will seed, archive, or clear
|
|
a Parquet. ``piker store anal`` and ``piker store ldshm`` are interactive
|
|
and potentially mutating; they are not audit commands.
|
|
|
|
Keep the disposable root private and do not rename, replace, or symlink
|
|
files under it while a helper is running. Stop the chart cleanly before
|
|
every audit or comparison phase.
|
|
|
|
Load The Xonsh Helpers
|
|
----------------------
|
|
|
|
Run from the repository root in the same xonsh which will launch the chart
|
|
or daemon::
|
|
|
|
source snippets/nativedb_backfill_audit.xsh
|
|
|
|
Choose a disposable root and copy only the provider configuration needed
|
|
for the case::
|
|
|
|
run_root = p'/tmp/piker-backfill/mnq-20260918-60s'
|
|
source_config = p'~/.config/piker'.expanduser()
|
|
bfq_init(run_root, source_config)
|
|
bfq_record_case(
|
|
run_root,
|
|
fqme='mnq.cme.20260918.ib',
|
|
period_s=60,
|
|
provider='ib',
|
|
symptom='gappy chart during reverse backfill',
|
|
bad_start_utc=None,
|
|
bad_end_utc=None,
|
|
)
|
|
|
|
``bfq_init()`` sets ``XDG_CONFIG_HOME`` in the current xonsh, so every
|
|
subsequent ``piker`` and ``pikerd`` child inherits the disposable config.
|
|
The effective storage path is under ``RUN_ROOT/xdg/piker/nativedb``.
|
|
It also records the exact repository root and ``piker`` executable in the
|
|
qualification marker, prepends that root to child ``PYTHONPATH``, and
|
|
verifies that the bound executable exposes ``piker store audit``. Audits
|
|
fail if that executable imports another tree.
|
|
|
|
The copied ``brokers.toml`` can contain credentials. Keep the run root
|
|
private and remove it manually after evidence is no longer needed.
|
|
|
|
Repository Preflight
|
|
--------------------
|
|
|
|
Before every daemon start, verify the code and config inherited by child
|
|
processes::
|
|
|
|
git status --short --branch
|
|
git rev-parse HEAD
|
|
python -c 'import piker; print(piker.__file__)'
|
|
python -c 'import os; print(os.environ["XDG_CONFIG_HOME"])'
|
|
|
|
Stop any daemon from the previous phase cleanly. Use a separate registry
|
|
address in the disposable ``conf.toml`` when a normal Piker stack remains
|
|
active.
|
|
|
|
Known-Bad Replay
|
|
----------------
|
|
|
|
Copy the original file; never move it out of normal storage::
|
|
|
|
fqme = 'mnq.cme.20260918.ib'
|
|
period_s = 60
|
|
source_parquet = (
|
|
p'~/.config/piker/nativedb'.expanduser()
|
|
/f'{fqme}.ohlcv{period_s}s.parquet'
|
|
)
|
|
bfq_seed(run_root, fqme, period_s, source_parquet)
|
|
bfq_audit(run_root, 'baseline', fqme, period_s)
|
|
|
|
The baseline phase writes::
|
|
|
|
evidence/baseline.json
|
|
evidence/baseline.parquet
|
|
evidence/baseline.manual.json
|
|
|
|
The JSON checksum must match the read-only snapshot, and the manual sidecar
|
|
binds that report to its phase and case. Comparison rechecks those bindings
|
|
before reading either phase. The report records raw schema, row and
|
|
timestamp counts, non-finite values, ordering, duplicate excess, canonical
|
|
index state, and every retained positive-gap detail. Files with invalid
|
|
Parquet encoding can not produce a JSON report, but ``--snapshot`` still
|
|
preserves their exact bytes before parse failure.
|
|
|
|
Launch the chart manually from the same shell and capture logs::
|
|
|
|
piker -l info chart mnq.cme.20260918.ib \
|
|
2>&1 | tee @(run_root / 'logs' / 'repair-chart.log')
|
|
|
|
After a clean stop::
|
|
|
|
bfq_audit(
|
|
run_root,
|
|
'repair',
|
|
fqme,
|
|
period_s,
|
|
require_structural=True,
|
|
)
|
|
bfq_compare(run_root, 'baseline', 'repair')
|
|
|
|
The comparison fails if structural validity regresses, any original
|
|
timestamp disappears, chronological endpoints shrink, timestamps become
|
|
duplicated or unordered, indexes cease to be canonical, or OHLCV becomes
|
|
non-finite. Provider replacement of matching seam rows is reported
|
|
separately as ``changed_common_timestamps`` for operator review.
|
|
|
|
Fresh Backfill
|
|
--------------
|
|
|
|
Yes, fresh qualification means saving the known-bad evidence and then
|
|
starting with no active Parquet in the disposable NativeDB. Archive only
|
|
the disposable file::
|
|
|
|
bfq_clear_for_fresh(run_root, fqme, period_s)
|
|
|
|
Confirm the printed active path is absent, launch the same chart, wait for
|
|
bounded provider exhaustion or the intended coverage, then stop cleanly::
|
|
|
|
bfq_audit(
|
|
run_root,
|
|
'fresh',
|
|
fqme,
|
|
period_s,
|
|
require_structural=True,
|
|
)
|
|
|
|
Do not compare ``baseline`` to ``fresh`` as a timestamp-preservation gate:
|
|
provider limits may intentionally produce a different initial extent.
|
|
Treat ``fresh`` as the baseline for restart and append qualification.
|
|
|
|
Restart And Append
|
|
------------------
|
|
|
|
Restart the same chart against the fresh persisted file, stop cleanly, and
|
|
record::
|
|
|
|
bfq_audit(
|
|
run_root,
|
|
'restart',
|
|
fqme,
|
|
period_s,
|
|
require_structural=True,
|
|
)
|
|
bfq_compare(run_root, 'fresh', 'restart')
|
|
|
|
Leave a subsequent run active long enough to append new samples, stop, and
|
|
record::
|
|
|
|
bfq_audit(
|
|
run_root,
|
|
'append',
|
|
fqme,
|
|
period_s,
|
|
require_structural=True,
|
|
)
|
|
bfq_compare(
|
|
run_root,
|
|
'restart',
|
|
'append',
|
|
require_newer=True,
|
|
)
|
|
|
|
The comparison JSON reports gaps added and removed. Expected closures stay
|
|
visible until a venue-aware classifier proves their session alignment.
|
|
|
|
Gap Layer Triage
|
|
----------------
|
|
|
|
Use the persisted report and chart together:
|
|
|
|
=============================== =========================================
|
|
Observation Suspected layer
|
|
=============================== =========================================
|
|
Parquet contiguous, chart gappy SHM hydration, sampling, or chart render
|
|
Parquet gap, provider has bars provider-delta or storage merge
|
|
Provider and Parquet both gappy provider omission or venue closure
|
|
Gap vanishes after restart SHM publication or cache lifecycle
|
|
Old timestamps disappear destructive persistence regression
|
|
Newest seam duplicates conflict or dedupe policy
|
|
=============================== =========================================
|
|
|
|
Always record exact UTC gap endpoints. The audit's numeric epoch is
|
|
authoritative; the ``*_utc`` fields are deterministic display values.
|
|
|
|
Expansion Matrix
|
|
----------------
|
|
|
|
Expand only after the fixed FQME passes fresh, restart, and append. Use a
|
|
new disposable root and case record for every row:
|
|
|
|
==================== ========== ========================================
|
|
Case Period Coverage model
|
|
==================== ========== ========================================
|
|
Original IB future 60 seconds CME maintenance and weekend closures
|
|
Original IB future 1 second high-frequency provider limits
|
|
Second CME future 60 seconds contract-specific history
|
|
IB equity 60 seconds regular overnight and weekend closure
|
|
IB FX 60 seconds 24x5 session behavior
|
|
Crypto spot/perp 60 seconds continuous-session control
|
|
Expired future 60 seconds history-only and expiry behavior
|
|
==================== ========== ========================================
|
|
|
|
A sourceable matrix can remain ordinary Python data in xonsh::
|
|
|
|
cases = [
|
|
{
|
|
'name': 'mnq-60s',
|
|
'fqme': 'mnq.cme.20260918.ib',
|
|
'period_s': 60,
|
|
'provider': 'ib',
|
|
},
|
|
{
|
|
'name': 'mnq-1s',
|
|
'fqme': 'mnq.cme.20260918.ib',
|
|
'period_s': 1,
|
|
'provider': 'ib',
|
|
},
|
|
]
|
|
for case in cases:
|
|
case_root = p'/tmp/piker-backfill' / case['name']
|
|
bfq_init(case_root, source_config)
|
|
bfq_record_case(
|
|
case_root,
|
|
fqme=case['fqme'],
|
|
period_s=case['period_s'],
|
|
provider=case['provider'],
|
|
symptom='manual expansion case',
|
|
)
|
|
|
|
This loop prepares isolated case roots only. Run the complete seed or
|
|
fresh, chart, audit, restart, append, and compare lifecycle in each root;
|
|
directory creation alone is not qualification.
|
|
|
|
The chart path may allocate companion 1-second and 60-second files. Audit
|
|
the selected case path explicitly and retain companion files as secondary
|
|
evidence; do not imply that the live request isolated one timeframe.
|
|
|
|
Interruption Qualification
|
|
--------------------------
|
|
|
|
After normal restart is stable, interrupt one phase with graceful Ctrl-C,
|
|
restart, and compare against the last clean evidence. Provider disconnect,
|
|
SIGTERM, and deterministic write-barrier failures come next. Avoid timing a
|
|
SIGKILL by hand; add an explicit failpoint before qualifying atomic replace
|
|
boundaries.
|
|
|
|
Result Interpretation
|
|
---------------------
|
|
|
|
``structural_ok`` means the raw persisted file satisfies durable schema,
|
|
finite values, positive unique ordered timestamps, canonical index, and
|
|
period-aligned gaps. ``gap_free`` means no positive gaps were observed when
|
|
at least two valid timestamps made cadence verifiable; otherwise it is
|
|
``null``. ``qualification_ok`` combines both and is intentionally strict.
|
|
|
|
A structurally valid session-market file can have ``gap_free = false``.
|
|
Those intervals require provider/session evidence before being marked as
|
|
expected closures. Unknown calendar state must stay unknown.
|