Compare commits

..

22 Commits

Author SHA1 Message Date
Gud Boi deaf7dd1ab Use `platformdirs` for `.config.get_app_dir()`
Replace hand-rolled `click`-based platform branching using
the much saner `platformdirs.user_config_path()`.

Deats,
- remove Windows/macOS/Unix `if/elif` platform dispatch
  (~25 lines) in favour of single `user_config_path()` call.
- move `_posixify()` inside `force_posix` branch since it's
  only used there.
- add `log.info()` reporting platform name and resolved dirs.

Also,
- drop now unneeded `sys` import.
- reformat `assert` in `repodir()` to multiline style.
- convert docstring from `r"""..."""` to `'''...'''` style.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-10 17:43:42 -04:00
Gud Boi 702aae2544 Drop bp from duration mismatch branch in `.ib.api.Client.bars()` 2026-03-10 17:09:49 -04:00
Gud Boi 44d54babeb Handle VNC reset-dialog in `vnc_click_hack()`
Add TAB + ENTER key presses after the `Ctrl+Alt+<key>` hotkey
combo to auto-confirm the "simulate a reset?" dialog that IB
gateway sometimes shows.

Deats,
- press `ISO_Enter` before click to dismiss any prior active
  dialog window.
- add post-hotkey loop sending `Tab` then `KP_Enter` with
  `asyncio.sleep()` delays to handle the confirmation dialog.
- add `asyncio` import.

Also,
- capture VNC connect error as `vnc_err` and log it instead of
  falling through to `try_xdo_manual()`.
- comment-out `try_xdo_manual()` fallback in VNC error path.
- reformat `client.press()` call to multiline style.
- reformat `RuntimeError` raise to multiline style with `!r`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-10 17:06:46 -04:00
Gud Boi b5a33e1217 Exclude crypto futes from `without_src` sym key
Extend the `col_sym_key` asset-type check in `start_backfill()`
to also exclude crypto-denominated futures (where `src` is
`'crypto_currency'` and `dst` is `'future'`) from the
`without_src=True` fqme path.

Also in `.brokers.binance` backend (it being the guilty culprit in the
discovery of this bug; and why i touched styling this code),

- reformat `make_sub()` fn sig to multiline style in
  `.binance.feed`.
- add backtick around `dict` in `make_sub()` docstring.
- reformat `or` conditionals to multiline style in
  `.binance.feed.get_mkt_info()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 22:13:06 -05:00
Gud Boi 796a831c6e Enable console log (from passed down `loglevel`) in `.tsp._history.manage_history()` 2026-03-06 09:26:34 -05:00
Gud Boi de81d1e905 Drop `Flume.feed`, it's unused yet causes import cycles.. 2026-03-06 09:26:34 -05:00
Gud Boi 170dd9794c Just warn on single-bar nulls instead of bping
Replace the debug breakpoint with a warning-log when a single-bar
null-segment is detected in `get_null_segs()`. This lets the gap
analysis continue while still alerting about the anomaly.

Deats,
- extract the 3-bar window (before, null, after) and calculate
  a `gap: pendulum.Interval` for the warning msg.
- comment-out the old breakpoint block for optional debugging as needed.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:26:34 -05:00
Gud Boi 599c36aba6 Lul, drop long unused poetry lock file 2026-03-06 09:26:34 -05:00
Gud Boi f174f79a1a Pin `pg` at latest official `0.14.0` release
Keep in masked GH sources lines for easy hackin against upstream
`master` branch when needed as well!
2026-03-06 09:26:34 -05:00
Gud Boi 9b284c2256 .ui._editors: log multiline styling and re-leveling 2026-03-06 09:26:34 -05:00
Gud Boi 59f2d46a97 .ui._lines: drop unused graphics-item import 2026-03-06 09:26:34 -05:00
Gud Boi c1b1e99693 Add batch-submit API for gap annotations
Introduce `AnnotCtl.add_batch()` and `serve_rc_annots()` batch
handler to submit 1000s of gaps in single IPC msg instead of
per-annot round-trips. Server builds `GapAnnotations` from specs
and handles vectorized timestamp-to-index lookups.

Deats,
- add `'cmd': 'batch'` handler in `serve_rc_annots()`
- vectorized timestamp lookup via `np.searchsorted()` + masking
- build `gap_specs: list[dict]` from rect+arrow specs client-side
- create single `GapAnnotations` item for all gaps server-side
- handle `GapAnnotations.reposition()` in redraw handler
- add profiling to batch path for perf measurement
- support optional individual arrows for A/B comparison

Also,
- refactor `markup_gaps()` to collect specs + single batch call
- add `no_qt_updates()` context mgr for batch render ops
- add profiling to annotation teardown path
- add `GapAnnotations` case to `rm_annot()` match block

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:26:34 -05:00
Gud Boi 24651d2326 Add a `GapAnnotations` path-renderer
For a ~1000x perf gain says ol' claudy, our boi who wrote this entire
patch! Bo

Introduce `GapAnnotations` in `.ui._annotate` for batch-rendering gap
rects/arrows instead of individual `QGraphicsItem` instances. Uses
upstream's `pyqtgraph.Qt.internals.PrimitiveArray` for rects and
a `QPainterPath` for arrows. This API-replicates our prior annotator's
in view shape-graphics but now using (what we're dubbing)
"single-array-multiple-graphics" tech much like our `.ui._curve`
extensions to `pg` B)

Impl deats,
- batch draw ~1000 gaps in single paint call vs 1000 items
- arrows render in scene coords to maintain pixel size on zoom
- add vectorized timestamp-to-index lookup for repositioning
- cache bounding rect, rebuild on `reposition()` calls
- match `SelectRect` + `ArrowItem` visual style/colors
- skip reposition when timeframe doesn't match gap's period

Other,
- fix typo in `LevelMarker` docstring: "graphich" -> "graphic"
- reflow docstring in `qgo_draw_markers()` to 67 char limit

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:26:34 -05:00
Gud Boi 2d00bb1024 Add info log for shm processing in `ldshm` CLI cmd
Log shm file name and detected period before null segment
processing to aid debugging.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:26:34 -05:00
Gud Boi dd40ad603f Bump to latest official `pyqtgraph` release 2026-03-06 09:26:34 -05:00
Gud Boi f2ace1b63b Use `ppfmt()` in `order_mode` since it's provided by `tractor` now 2026-03-06 09:22:36 -05:00
Gud Boi 9010f9c7ab Augment `.ib.symbols` search with more logging
Refactor `open_symbol_search()` to use `partial()` for nursery task
spawning and add detailed query->results logging via `ppfmt()`.

Deats,
- change `extend_results()` to accept `target` callable +
  `pattern` + `**kwargs` and invoke inside, instead of receiving
  a pre-called awaitable; use `partial()` to pass args.
- add `ppfmt()` formatted logging of search query params and
  results including client class + method repr.
- change `print()` -> `log.exception()` for `Lagged` overrun.
- bump `upto=5` -> `upto=10` for `search_symbols()` call.

Also for styling,
- add type some missing type annots.
- add multiline style to `or` conditionals in pattern check.
- reformat log msgs to multiline style throughout.
- use `ppfmt()` for fuzzy match debug log.
- rename nursery `sn` -> `tn`.
- add TODO comment about `assert 0` hang.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:22:36 -05:00
Gud Boi 89a145113c Handle `str`-errors in `.ib.broker` trade events
Add `isinstance()` dispatch for the `'error'` event case in
`deliver_trade_events()` to handle `ib_async` sometimes emitting plain
`str` error items instead of the previously expected `dict`.

Deats,
- add `isinstance(err, dict)` branch for the standard case with
  `error_code`, `reason`, and `reqid` fields.
- add `isinstance(err, str)` branch to parse error strings of the
  form `'[code 104] connection failed'` into `code` and `reason`.
- set `reqid: str = '<unknown>'` for string-form errors since
  there's no request ID available.
- update `err` type annot to `dict|str`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:22:36 -05:00
Gud Boi ec4db30cdc Handle valid null frames and 0-bar cases in backfill
Add guards for empty-array and zero-bar-diff cases in the TSP backfill
loops to avoid crashes and allow graceful loop termination.

In `maybe_fill_null_segments()`,
- add `array.size == 0` guard in `maybe_fill_null_segments()` to detect
  valid (venue closure) gaps from the backend; add a warning + bp
  + break for this case.
- add TODO that we should likely be filling nulls with the close price
  for the gap's duration.

In `start_backfill()`,
- expand the "0 bars after diff" warning msg with
  `backfill_until_dt` and `end_dt_param` context.
- mask the  `await tractor.pause()` and add a `break` to avoid blocking
  the backfill loop.

(this commit msg was generated in some part by
[`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:22:36 -05:00
Gud Boi 2a394dba03 Warn instead of raise on `start_dt`-trimmed frames
Downgrade the `start_dt`-trimming check in `open_history_client()`
from a `RuntimeError` raise to a warning log, allowing the caller
to still receive a (shorter) frame of bars (though we may need to still
specially handle such cases in the backfiller's biz logic layer).

Deats,
- add `trimmed_bars.size` guard to skip check on empty results.
- change condition to `>=` and log a warning with the short-frame
  size instead of raising.
- comment-out `raise RuntimeError` and breakpoint for future
  removal once confident.
- add docstring-style comment on `start_dt=` kwarg noting that
  `Client.bars()` doesn't truly support it (uses duration-style
  queries internally).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:22:36 -05:00
Gud Boi 19f16e1df3 Handle ambiguous futes contracts in `get_fute()`
Use (the only available in `ib_async`) `returnAll=True` in
`qualifyContractsAsync()` calls within `get_fute()` and handle the case
where IB returns a list of ambiguous contract matches instead of
a single result.

Deats,
- add `returnAll=True` to both `ContFuture` and `Future`
  qualification calls.
- add `isinstance(con, list)` check after unpacking first result
  to detect ambiguous contract sets.
- log warning with input params and matched contracts when
  ambiguous.
- update return type annot to `Contract|list[Contract]`.

Also,
- handle list-of-contracts case in `find_contracts()` by unpacking
  `*contracts` into the `qualifyContractsAsync()` call.
- reformat `qualifyContractsAsync()` calls to multiline style.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:22:36 -05:00
Gud Boi 3adb0d8b9d Fall back to `Contract.exchange` in `has_holiday()`
Use `con.exchange` as fallback when `con.primaryExchange` is empty
in `has_holiday()` to handle contracts like futures that don't
always set a `primaryExchange`.

Deats,
- extract `con: Contract` from `con_deats.contract` for reuse.
- use `con.primaryExchange or con.exchange` to ensure a valid
  exchange code is always passed to the calendar lookup.
- add `Contract` to `TYPE_CHECKING` imports.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-06 09:22:36 -05:00
5 changed files with 98 additions and 61 deletions

View File

@ -203,9 +203,13 @@ async def stream_messages(
yield 'trade', piker_quote
def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
def make_sub(
pairs: list[str],
sub_name: str,
uid: int,
) -> dict[str, str]:
'''
Create a request subscription packet dict.
Create a request subscription packet `dict`.
- spot:
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
@ -332,7 +336,8 @@ async def get_mkt_info(
# TODO: handle coinm futes which have a margin asset that
# is some crypto token!
# https://binance-docs.github.io/apidocs/delivery/en/#exchange-information
or 'btc' in venue_lower
or
'btc' in venue_lower
):
return None
@ -343,12 +348,14 @@ async def get_mkt_info(
if (
venue
and 'spot' not in venue_lower
and
'spot' not in venue_lower
# XXX: catch all in case user doesn't know which
# venue they want (usdtm vs. coinm) and we can choose
# a default (via config?) once we support coin-m APIs.
or 'perp' in venue_lower
or
'perp' in venue_lower
):
if not mkt_mode:
mkt_mode: str = f'{venue_lower}_futes'

View File

@ -20,6 +20,7 @@ runnable script-programs.
'''
from __future__ import annotations
import asyncio
from datetime import ( # noqa
datetime,
date,
@ -140,7 +141,8 @@ async def data_reset_hack(
except (
OSError, # no VNC server avail..
PermissionError, # asyncvnc pw fail..
):
) as _vnc_err:
vnc_err = _vnc_err
try:
import i3ipc # noqa (since a deps dynamic check)
except ModuleNotFoundError:
@ -166,14 +168,22 @@ async def data_reset_hack(
# localhost but no vnc-client or it borked..
else:
try_xdo_manual(client)
log.error(
'VNC CLICK HACK FAILE with,\n'
f'{vnc_err!r}\n'
)
# breakpoint()
# try_xdo_manual(client)
case 'i3ipc_xdotool':
try_xdo_manual(client)
# i3ipc_xdotool_manual_click_hack()
case _ as tech:
raise RuntimeError(f'{tech} is not supported for reset tech!?')
raise RuntimeError(
f'{tech!r} is not supported for reset tech!?'
)
# we don't really need the ``xdotool`` approach any more B)
return True
@ -269,10 +279,35 @@ async def vnc_click_hack(
500,
)
)
# in case a prior dialog win is open/active.
await client.press('ISO_Enter')
# ensure the ib-gw window is active
await client.click(MOUSE_BUTTON_LEFT)
# send the hotkeys combo B)
await client.press('Ctrl', 'Alt', key) # keys are stacked
await client.press(
'Ctrl',
'Alt',
key,
) # NOTE, keys are stacked
# XXX, sometimes a dialog asking if you want to "simulate
# a reset" will show, in which case we want to select
# "Yes" (by tabbing) and then hit enter.
iters: int = 1
delay: float = 0.3
await asyncio.sleep(delay)
for i in range(iters):
log.info(f'Sending TAB {i}')
await client.press('Tab')
await asyncio.sleep(delay)
for i in range(iters):
log.info(f'Sending ENTER {i}')
await client.press('KP_Enter')
await asyncio.sleep(delay)
def i3ipc_fin_wins_titled(

View File

@ -561,7 +561,7 @@ class Client:
# f'Recursing for more bars:\n'
)
# XXX, debug!
breakpoint()
# breakpoint()
# XXX ? TODO? recursively try to re-request?
# => i think *NO* right?
#

View File

@ -19,7 +19,6 @@ Platform configuration (files) mgmt.
"""
import platform
import sys
import os
import shutil
from typing import (
@ -29,6 +28,7 @@ from typing import (
from pathlib import Path
from bidict import bidict
import platformdirs
import tomlkit
try:
import tomllib
@ -41,7 +41,7 @@ from .log import get_logger
log = get_logger('broker-config')
# XXX NOTE: taken from `click`
# XXX NOTE: orig impl was taken from `click`
# |_https://github.com/pallets/click/blob/main/src/click/utils.py#L449
#
# (since apparently they have some super weirdness with SIGINT and
@ -54,44 +54,21 @@ def get_app_dir(
force_posix: bool = False,
) -> str:
r"""Returns the config folder for the application. The default behavior
'''
Returns the config folder for the application. The default behavior
is to return whatever is most appropriate for the operating system.
To give you an idea, for an app called ``"Foo Bar"``, something like
the following folders could be returned:
----
NOTE, below is originally from `click` impl fn, we can prolly remove?
----
Mac OS X:
``~/Library/Application Support/Foo Bar``
Mac OS X (POSIX):
``~/.foo-bar``
Unix:
``~/.config/foo-bar``
Unix (POSIX):
``~/.foo-bar``
Win XP (roaming):
``C:\Documents and Settings\<user>\Local Settings\Application Data\Foo``
Win XP (not roaming):
``C:\Documents and Settings\<user>\Application Data\Foo Bar``
Win 7 (roaming):
``C:\Users\<user>\AppData\Roaming\Foo Bar``
Win 7 (not roaming):
``C:\Users\<user>\AppData\Local\Foo Bar``
.. versionadded:: 2.0
:param app_name: the application name. This should be properly capitalized
and can contain whitespace.
:param roaming: controls if the folder should be roaming or not on Windows.
Has no affect otherwise.
:param force_posix: if this is set to `True` then on any POSIX system the
folder will be stored in the home folder with a leading
dot instead of the XDG config home or darwin's
application support folder.
"""
def _posixify(name):
return "-".join(name.split()).lower()
'''
# NOTE: for testing with `pytest` we leverage the `tmp_dir`
# fixture to generate (and clean up) a test-request-specific
# directory for isolated configuration files such that,
@ -117,23 +94,30 @@ def get_app_dir(
# assert testdirpath.exists(), 'piker test harness might be borked!?'
# app_name = str(testdirpath)
if platform.system() == 'Windows':
key = "APPDATA" if roaming else "LOCALAPPDATA"
folder = os.environ.get(key)
if folder is None:
folder = os.path.expanduser("~")
return os.path.join(folder, app_name)
os_name: str = platform.system()
conf_dir: Path = platformdirs.user_config_path()
app_dir: Path = conf_dir / app_name
# ?TODO, from `click`; can remove?
if force_posix:
def _posixify(name):
return "-".join(name.split()).lower()
return os.path.join(
os.path.expanduser("~/.{}".format(_posixify(app_name))))
if sys.platform == "darwin":
return os.path.join(
os.path.expanduser("~/Library/Application Support"), app_name
os.path.expanduser(
"~/.{}".format(
_posixify(app_name)
)
return os.path.join(
os.environ.get("XDG_CONFIG_HOME", os.path.expanduser("~/.config")),
_posixify(app_name),
)
)
log.info(
f'Using user config directory,\n'
f'platform.system(): {os_name!r}\n'
f'conf_dir: {conf_dir!r}\n'
f'app_dir: {conf_dir!r}\n'
)
return app_dir
_click_config_dir: Path = Path(get_app_dir('piker'))
@ -250,7 +234,9 @@ def repodir() -> Path:
repodir: Path = Path(os.environ.get('GITHUB_WORKSPACE'))
confdir: Path = repodir / 'config'
assert confdir.is_dir(), f'{confdir} DNE, {repodir} is likely incorrect!'
assert confdir.is_dir(), (
f'{confdir} DNE, {repodir} is likely incorrect!'
)
return repodir

View File

@ -739,12 +739,21 @@ async def start_backfill(
# including the dst[/src] source asset token. SO,
# 'tsla.nasdaq.ib' over 'tsla/usd.nasdaq.ib' for
# historical reasons ONLY.
if mkt.dst.atype not in {
if (
mkt.dst.atype not in {
'crypto',
'crypto_currency',
'fiat', # a "forex pair"
'perpetual_future', # stupid "perps" from cex land
}:
}
and not (
mkt.src.atype == 'crypto_currency'
and
mkt.dst.atype in {
'future',
}
)
):
col_sym_key: str = mkt.get_fqme(
delim_char='',
without_src=True,