Compare commits
No commits in common. "main" and "py311_ib_fix" have entirely different histories.
main
...
py311_ib_f
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(chmod:*)",
|
||||
"Bash(/tmp/piker_commits.txt)",
|
||||
"Bash(python:*)"
|
||||
],
|
||||
"deny": [],
|
||||
"ask": []
|
||||
}
|
||||
}
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
---
|
||||
name: commit-msg
|
||||
description: >
|
||||
Generate piker-style git commit messages from
|
||||
staged changes or prompt input, following the
|
||||
style guide learned from 500 repo commits.
|
||||
argument-hint: "[optional-scope-or-description]"
|
||||
disable-model-invocation: true
|
||||
allowed-tools: Bash(git *), Read, Grep, Glob, Write
|
||||
---
|
||||
|
||||
## Current staged changes
|
||||
!`git diff --staged --stat`
|
||||
|
||||
## Recent commit style reference
|
||||
!`git log --oneline -10`
|
||||
|
||||
# Piker Git Commit Message Generator
|
||||
|
||||
Generate a commit message from the staged diff above
|
||||
following the piker project's conventions (learned from
|
||||
analyzing 500 repo commits).
|
||||
|
||||
If `$ARGUMENTS` is provided, use it as scope or
|
||||
description context for the commit message.
|
||||
|
||||
For the full style guide with verb frequencies,
|
||||
section markers, abbreviations, piker-specific terms,
|
||||
and examples, see
|
||||
[style-guide-reference.md](./style-guide-reference.md).
|
||||
|
||||
## Quick Reference
|
||||
|
||||
- **Subject**: ~50 chars, present tense verb, use
|
||||
backticks for code refs
|
||||
- **Body**: only for complex/multi-file changes,
|
||||
67 char line max
|
||||
- **Section markers**: Also, / Deats, / Other,
|
||||
- **Bullets**: use `-` style
|
||||
- **Tone**: technical but casual (piker style)
|
||||
|
||||
## Claude-code Footer
|
||||
|
||||
When the written **patch** was assisted by
|
||||
claude-code, include:
|
||||
|
||||
```
|
||||
(this patch was generated in some part by [`claude-code`][claude-code-gh])
|
||||
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||
```
|
||||
|
||||
When only the **commit msg** was written by
|
||||
claude-code (human wrote the patch), use:
|
||||
```
|
||||
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
|
||||
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||
```
|
||||
|
||||
## Output Instructions
|
||||
|
||||
When generating a commit message:
|
||||
|
||||
1. Analyze the staged diff (injected above via
|
||||
dynamic context) to understand all changes.
|
||||
2. If `$ARGUMENTS` provides a scope (e.g.,
|
||||
`.ib.feed`) or description, incorporate it into
|
||||
the subject line.
|
||||
3. Write the subject line following verb + backtick
|
||||
conventions from the
|
||||
[style guide](./style-guide-reference.md).
|
||||
4. Add body only for multi-file or complex changes.
|
||||
5. Write the message to a file in the repo's
|
||||
`.claude/` subdir with filename format:
|
||||
`<timestamp>_<first-7-chars-of-last-commit-hash>_commit_msg.md`
|
||||
where `<timestamp>` is from `date --iso-8601=seconds`.
|
||||
Also write a copy to
|
||||
`.claude/git_commit_msg_LATEST.md`
|
||||
(overwrite if exists).
|
||||
|
||||
---
|
||||
|
||||
**Analysis date:** 2026-01-27
|
||||
**Commits analyzed:** 500 from piker repository
|
||||
**Maintained by:** Tyler Goodlet
|
||||
|
|
@ -1,262 +0,0 @@
|
|||
# Piker Git Commit Message Style Guide
|
||||
|
||||
Learned from analyzing 500 commits from the piker repository.
|
||||
|
||||
## Subject Line Rules
|
||||
|
||||
### Length
|
||||
- Target: ~50 characters (avg: 50.5 chars)
|
||||
- Maximum: 67 chars (hard limit, though historical max: 146)
|
||||
- Keep concise and descriptive
|
||||
|
||||
### Structure
|
||||
- Use present tense verbs (Add, Drop, Fix, Move, etc.)
|
||||
- 65.6% of commits use backticks for code references
|
||||
- 33.0% use colon notation (`module.file:` prefix or `: ` separator)
|
||||
|
||||
### Opening Verbs (by frequency)
|
||||
Primary verbs to use:
|
||||
- **Add** (8.4%) - New features, files, functionality
|
||||
- **Drop** (3.2%) - Remove features, dependencies, code
|
||||
- **Fix** (2.2%) - Bug fixes, corrections
|
||||
- **Use** (2.2%) - Switch to different approach/tool
|
||||
- **Port** (2.0%) - Migrate code, adapt from elsewhere
|
||||
- **Move** (2.0%) - Relocate code, refactor structure
|
||||
- **Always** (1.8%) - Enforce consistent behavior
|
||||
- **Factor** (1.6%) - Refactoring, code organization
|
||||
- **Bump** (1.6%) - Version/dependency updates
|
||||
- **Update** (1.4%) - Modify existing functionality
|
||||
- **Adjust** (1.0%) - Fine-tune, tweak behavior
|
||||
- **Change** (1.0%) - Modify behavior or structure
|
||||
|
||||
Casual/informal verbs (used occasionally):
|
||||
- **Woops,** (1.4%) - Fixing mistakes
|
||||
- **Lul,** (0.6%) - Humorous corrections
|
||||
|
||||
### Code References
|
||||
Use backticks heavily for:
|
||||
- **Module/package names**: `tractor`, `pikerd`, `polars`, `ruff`
|
||||
- **Data types**: `dict`, `float`, `str`, `None`
|
||||
- **Classes**: `MktPair`, `Asset`, `Position`, `Account`, `Flume`
|
||||
- **Functions**: `dedupe()`, `push()`, `get_client()`, `norm_trade()`
|
||||
- **File paths**: `.tsp`, `.fqme`, `brokers.toml`, `conf.toml`
|
||||
- **CLI flags**: `--pdb`
|
||||
- **Error types**: `NoData`
|
||||
- **Tools**: `uv`, `uv sync`, `httpx`, `numpy`
|
||||
|
||||
### Colon Usage Patterns
|
||||
1. **Module prefix**: `.ib.feed: trim bars frame to start_dt`
|
||||
2. **Separator**: `Add support: new feature description`
|
||||
|
||||
### Tone
|
||||
- Technical but casual (use XD, lol, .., Woops, Lul when appropriate)
|
||||
- Direct and concise
|
||||
- Question marks rare (1.4%)
|
||||
- Exclamation marks rare (1.4%)
|
||||
|
||||
## Body Structure
|
||||
|
||||
### Body Frequency
|
||||
- 56.0% of commits have empty bodies (one-line commits are common)
|
||||
- Use body for complex changes requiring explanation
|
||||
|
||||
### Bullet Lists
|
||||
- Prefer `-` bullets (16.2% of commits)
|
||||
- Rarely use `*` bullets (1.6%)
|
||||
- Indent continuation lines appropriately
|
||||
|
||||
### Section Markers (in order of frequency)
|
||||
Use these to organize complex commit bodies:
|
||||
|
||||
1. **Also,** (most common, 26 occurrences)
|
||||
- Additional changes, side effects, related updates
|
||||
- Example:
|
||||
```
|
||||
Main change described in subject.
|
||||
|
||||
Also,
|
||||
- related change 1
|
||||
- related change 2
|
||||
```
|
||||
|
||||
2. **Deats,** (8 occurrences)
|
||||
- Implementation details
|
||||
- Technical specifics
|
||||
|
||||
3. **Further,** (4 occurrences)
|
||||
- Additional context or future considerations
|
||||
|
||||
4. **Other,** (3 occurrences)
|
||||
- Miscellaneous related changes
|
||||
|
||||
5. **Notes,** **TODO,** (rare, 1 each)
|
||||
- Special annotations when needed
|
||||
|
||||
### Line Length
|
||||
- Body lines: 67 character maximum
|
||||
- Break longer lines appropriately
|
||||
|
||||
## Language Patterns
|
||||
|
||||
### Common Abbreviations (by frequency)
|
||||
Use these freely in commit bodies:
|
||||
- **msg** (29) - message
|
||||
- **mod** (15) - module
|
||||
- **vs** (14) - versus
|
||||
- **impl** (12) - implementation
|
||||
- **deps** (11) - dependencies
|
||||
- **var** (6) - variable
|
||||
- **ctx** (6) - context
|
||||
- **bc** (5) - because
|
||||
- **obvi** (4) - obviously
|
||||
- **ep** (4) - endpoint
|
||||
- **tn** (4) - task name
|
||||
- **rn** (3) - right now
|
||||
- **sig** (3) - signal/signature
|
||||
- **env** (3) - environment
|
||||
- **tho** (3) - though
|
||||
- **fn** (2) - function
|
||||
- **iface** (2) - interface
|
||||
- **prolly** (2) - probably
|
||||
|
||||
Less common but acceptable:
|
||||
- **dne**, **osenv**, **gonna**, **wtf**
|
||||
|
||||
### Tone Indicators
|
||||
- **..** (77 occurrences) - Ellipsis for trailing thoughts
|
||||
- **XD** (17) - Expression of humor/irony
|
||||
- **lol** (1) - Rare, use sparingly
|
||||
|
||||
### Informal Patterns
|
||||
- Casual contractions okay: Don't, won't
|
||||
- Lowercase starts acceptable for file prefixes
|
||||
- Direct, conversational tone
|
||||
|
||||
## Special Patterns
|
||||
|
||||
### Module/File Prefixes
|
||||
Common in piker commits (33.0% use colons):
|
||||
- `.ib.feed: description`
|
||||
- `.ui._remote_ctl: description`
|
||||
- `.data.tsp: description`
|
||||
- `.accounting: description`
|
||||
|
||||
### Merge Commits
|
||||
- 4.4% of commits (standard git merges)
|
||||
- Not a primary pattern to emulate
|
||||
|
||||
### External References
|
||||
- GitHub links occasionally used (13 total)
|
||||
- File:line references not used (0 occurrences)
|
||||
- No WIP commits in analyzed set
|
||||
|
||||
### Claude-code Footer
|
||||
When the written **patch** was assisted by claude-code,
|
||||
include:
|
||||
|
||||
```
|
||||
(this patch was generated in some part by [`claude-code`][claude-code-gh])
|
||||
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||
```
|
||||
|
||||
When only the **commit msg** was written by claude-code
|
||||
(human wrote the patch), use:
|
||||
|
||||
```
|
||||
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
|
||||
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||
```
|
||||
|
||||
## Piker-Specific Terms
|
||||
|
||||
### Core Components
|
||||
- `pikerd` - piker daemon
|
||||
- `brokerd` - broker daemon
|
||||
- `tractor` - actor framework used
|
||||
- `.tsp` - time series protocol/module
|
||||
- `.fqme` - fully qualified market endpoint
|
||||
|
||||
### Data Structures
|
||||
- `MktPair` - market pair
|
||||
- `Asset` - asset representation
|
||||
- `Position` - trading position
|
||||
- `Account` - account data
|
||||
- `Flume` - data stream
|
||||
- `SymbologyCache` - symbol caching
|
||||
|
||||
### Common Functions
|
||||
- `dedupe()` - deduplication
|
||||
- `push()` - data pushing
|
||||
- `get_client()` - client retrieval
|
||||
- `norm_trade()` - trade normalization
|
||||
- `open_trade_ledger()` - ledger opening
|
||||
- `markup_gaps()` - gap marking
|
||||
- `get_null_segs()` - null segment retrieval
|
||||
- `remote_annotate()` - remote annotation
|
||||
|
||||
### Brokers & Integrations
|
||||
- `binance` - Binance integration
|
||||
- `.ib` - Interactive Brokers
|
||||
- `bs_mktid` - broker-specific market ID
|
||||
- `reqid` - request ID
|
||||
|
||||
### Configuration
|
||||
- `brokers.toml` - broker configuration
|
||||
- `conf.toml` - general configuration
|
||||
|
||||
### Development Tools
|
||||
- `ruff` - Python linter
|
||||
- `uv` / `uv sync` - package manager
|
||||
- `--pdb` - debugger flag
|
||||
- `pdbp` - debugger
|
||||
- `asyncvnc` / `pyvnc` - VNC libraries
|
||||
- `httpx` - HTTP client
|
||||
- `polars` - dataframe library
|
||||
- `rapidfuzz` - fuzzy matching
|
||||
- `numpy` - numerical library
|
||||
- `trio` - async framework
|
||||
- `asyncio` - async framework
|
||||
- `xonsh` - shell
|
||||
|
||||
## Examples
|
||||
|
||||
### Simple one-liner
|
||||
```
|
||||
Add `MktPair.fqme` property for symbol resolution
|
||||
```
|
||||
|
||||
### With module prefix
|
||||
```
|
||||
.ib.feed: trim bars frame to `start_dt`
|
||||
```
|
||||
|
||||
### Casual fix
|
||||
```
|
||||
Woops, compare against first-dt in `.ib.feed` bars frame
|
||||
```
|
||||
|
||||
### With body using "Also,"
|
||||
```
|
||||
Drop `poetry` for `uv` in dev workflow
|
||||
|
||||
Also,
|
||||
- update deps in `pyproject.toml`
|
||||
- add `uv sync` to CI pipeline
|
||||
- remove old `poetry.lock`
|
||||
```
|
||||
|
||||
### With implementation details
|
||||
```
|
||||
Factor position tracking into `Position` dataclass
|
||||
|
||||
Deats,
|
||||
- move calc logic from `brokerd` to `.accounting`
|
||||
- add `norm_trade()` helper for broker normalization
|
||||
- use `MktPair.fqme` for consistent symbol refs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Analysis date:** 2026-01-27
|
||||
**Commits analyzed:** 500 from piker repository
|
||||
**Maintained by:** Tyler Goodlet
|
||||
|
|
@ -1,171 +0,0 @@
|
|||
---
|
||||
name: piker-profiling
|
||||
description: >
|
||||
Piker's `Profiler` API for measuring performance
|
||||
across distributed actor systems. Apply when
|
||||
adding profiling, debugging perf regressions, or
|
||||
optimizing hot paths in piker code.
|
||||
user-invocable: false
|
||||
---
|
||||
|
||||
# Piker Profiling Subsystem
|
||||
|
||||
Skill for using `piker.toolz.profile.Profiler` to
|
||||
measure performance across distributed actor systems.
|
||||
|
||||
## Core Profiler API
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from piker.toolz.profile import (
|
||||
Profiler,
|
||||
pg_profile_enabled,
|
||||
ms_slower_then,
|
||||
)
|
||||
|
||||
profiler = Profiler(
|
||||
msg='<description of profiled section>',
|
||||
disabled=False, # IMPORTANT: enable explicitly!
|
||||
ms_threshold=0.0, # show all timings
|
||||
)
|
||||
|
||||
# do work
|
||||
some_operation()
|
||||
profiler('step 1 complete')
|
||||
|
||||
# more work
|
||||
another_operation()
|
||||
profiler('step 2 complete')
|
||||
|
||||
# prints on exit:
|
||||
# > Entering <description of profiled section>
|
||||
# step 1 complete: 12.34, tot:12.34
|
||||
# step 2 complete: 56.78, tot:69.12
|
||||
# < Exiting <description>, total: 69.12 ms
|
||||
```
|
||||
|
||||
### Default Behavior Gotcha
|
||||
|
||||
**CRITICAL:** Profiler is disabled by default in
|
||||
many contexts!
|
||||
|
||||
```python
|
||||
# BAD: might not print anything!
|
||||
profiler = Profiler(msg='my operation')
|
||||
|
||||
# GOOD: explicit enable
|
||||
profiler = Profiler(
|
||||
msg='my operation',
|
||||
disabled=False, # force enable!
|
||||
ms_threshold=0.0, # show all steps
|
||||
)
|
||||
```
|
||||
|
||||
### Profiler Output Format
|
||||
|
||||
```
|
||||
> Entering <msg>
|
||||
<label 1>: <delta_ms>, tot:<cumulative_ms>
|
||||
<label 2>: <delta_ms>, tot:<cumulative_ms>
|
||||
...
|
||||
< Exiting <msg>, total time: <total_ms> ms
|
||||
```
|
||||
|
||||
**Reading the output:**
|
||||
- `delta_ms` = time since previous checkpoint
|
||||
- `cumulative_ms` = time since profiler creation
|
||||
- Final total = end-to-end time
|
||||
|
||||
## Profiling Distributed Systems
|
||||
|
||||
Piker runs across multiple processes (actors). Each
|
||||
actor has its own log output.
|
||||
|
||||
### Common piker actors
|
||||
- `pikerd` - main daemon process
|
||||
- `brokerd` - broker connection actor
|
||||
- `chart` - UI/graphics actor
|
||||
- Client scripts - analysis/annotation clients
|
||||
|
||||
### Cross-Actor Profiling Strategy
|
||||
|
||||
1. Add `Profiler` on **both** client and server
|
||||
2. Correlate timestamps from each actor's output
|
||||
3. Calculate IPC overhead = total - (client + server
|
||||
processing)
|
||||
|
||||
**Example correlation:**
|
||||
|
||||
Client console:
|
||||
```
|
||||
> Entering markup_gaps() for 1285 gaps
|
||||
initial redraw: 0.20ms, tot:0.20
|
||||
built annotation specs: 256.48ms, tot:256.68
|
||||
batch IPC call complete: 119.26ms, tot:375.94
|
||||
final redraw: 0.07ms, tot:376.02
|
||||
< Exiting markup_gaps(), total: 376.04ms
|
||||
```
|
||||
|
||||
Server console (chart actor):
|
||||
```
|
||||
> Entering Batch annotate 1285 gaps
|
||||
`np.searchsorted()` complete!: 0.81ms, tot:0.81
|
||||
`time_to_row` creation: 98.45ms, tot:99.28
|
||||
created GapAnnotations item: 2.98ms, tot:102.26
|
||||
< Exiting Batch annotate, total: 104.15ms
|
||||
```
|
||||
|
||||
**Analysis:**
|
||||
- Total client time: 376ms
|
||||
- Server processing: 104ms
|
||||
- IPC overhead + client spec building: 272ms
|
||||
- Bottleneck: client-side spec building (256ms)
|
||||
|
||||
## Integration with PyQtGraph
|
||||
|
||||
Some piker modules integrate with `pyqtgraph`'s
|
||||
profiling:
|
||||
|
||||
```python
|
||||
from piker.toolz.profile import (
|
||||
Profiler,
|
||||
pg_profile_enabled,
|
||||
ms_slower_then,
|
||||
)
|
||||
|
||||
profiler = Profiler(
|
||||
msg='Curve.paint()',
|
||||
disabled=not pg_profile_enabled(),
|
||||
ms_threshold=ms_slower_then,
|
||||
)
|
||||
```
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
**Typical timings:**
|
||||
- IPC round-trip (local actors): 1-10ms
|
||||
- NumPy binary search (10k array): <1ms
|
||||
- Dict building (1k items, simple): 1-5ms
|
||||
- Qt redraw trigger: 0.1-1ms
|
||||
- Scene item removal (100s items): 10-50ms
|
||||
|
||||
**Red flags:**
|
||||
- Linear array scan per item: 50-100ms+ for 1k
|
||||
- Dict comprehension with struct array: 50-100ms
|
||||
- Individual Qt item creation: 5ms per item
|
||||
|
||||
## References
|
||||
|
||||
- `piker/toolz/profile.py` - Profiler impl
|
||||
- `piker/ui/_curve.py` - FlowGraphic paint profiling
|
||||
- `piker/ui/_remote_ctl.py` - IPC handler profiling
|
||||
- `piker/tsp/_annotate.py` - Client-side profiling
|
||||
|
||||
See [patterns.md](patterns.md) for detailed
|
||||
profiling patterns and debugging techniques.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-01-31*
|
||||
*Session: Batch gap annotation optimization*
|
||||
|
|
@ -1,228 +0,0 @@
|
|||
# Profiling Patterns
|
||||
|
||||
Detailed profiling patterns for use with
|
||||
`piker.toolz.profile.Profiler`.
|
||||
|
||||
## Pattern: Function Entry/Exit
|
||||
|
||||
```python
|
||||
async def my_function():
|
||||
profiler = Profiler(
|
||||
msg='my_function()',
|
||||
disabled=False,
|
||||
ms_threshold=0.0,
|
||||
)
|
||||
|
||||
step1()
|
||||
profiler('step1')
|
||||
|
||||
step2()
|
||||
profiler('step2')
|
||||
|
||||
# auto-prints on exit
|
||||
```
|
||||
|
||||
## Pattern: Loop Iterations
|
||||
|
||||
```python
|
||||
# DON'T profile inside tight loops (overhead!)
|
||||
for i in range(1000):
|
||||
profiler(f'iteration {i}') # NO!
|
||||
|
||||
# DO profile around loops
|
||||
profiler = Profiler(msg='processing 1000 items')
|
||||
for i in range(1000):
|
||||
process(item[i])
|
||||
profiler('processed all items')
|
||||
```
|
||||
|
||||
## Pattern: Conditional Profiling
|
||||
|
||||
```python
|
||||
# only profile when investigating specific issue
|
||||
DEBUG_REPOSITION = True
|
||||
|
||||
def reposition(self, array):
|
||||
if DEBUG_REPOSITION:
|
||||
profiler = Profiler(
|
||||
msg='GapAnnotations.reposition()',
|
||||
disabled=False,
|
||||
)
|
||||
|
||||
# ... do work
|
||||
|
||||
if DEBUG_REPOSITION:
|
||||
profiler('completed reposition')
|
||||
```
|
||||
|
||||
## Pattern: Teardown/Cleanup Profiling
|
||||
|
||||
```python
|
||||
try:
|
||||
# ... main work
|
||||
pass
|
||||
finally:
|
||||
profiler = Profiler(
|
||||
msg='Annotation teardown',
|
||||
disabled=False,
|
||||
ms_threshold=0.0,
|
||||
)
|
||||
|
||||
cleanup_resources()
|
||||
profiler('resources cleaned')
|
||||
|
||||
close_connections()
|
||||
profiler('connections closed')
|
||||
```
|
||||
|
||||
## Pattern: Distributed IPC Profiling
|
||||
|
||||
### Server-side (chart actor)
|
||||
|
||||
```python
|
||||
# piker/ui/_remote_ctl.py
|
||||
@tractor.context
|
||||
async def remote_annotate(ctx):
|
||||
async with ctx.open_stream() as stream:
|
||||
async for msg in stream:
|
||||
profiler = Profiler(
|
||||
msg=f'Batch annotate {n} gaps',
|
||||
disabled=False,
|
||||
ms_threshold=0.0,
|
||||
)
|
||||
|
||||
result = await handle_request(msg)
|
||||
profiler('request handled')
|
||||
|
||||
await stream.send(result)
|
||||
profiler('result sent')
|
||||
```
|
||||
|
||||
### Client-side (analysis script)
|
||||
|
||||
```python
|
||||
# piker/tsp/_annotate.py
|
||||
async def markup_gaps(...):
|
||||
profiler = Profiler(
|
||||
msg=f'markup_gaps() for {n} gaps',
|
||||
disabled=False,
|
||||
ms_threshold=0.0,
|
||||
)
|
||||
|
||||
await actl.redraw()
|
||||
profiler('initial redraw')
|
||||
|
||||
specs = build_specs(gaps)
|
||||
profiler('built annotation specs')
|
||||
|
||||
# IPC round-trip!
|
||||
result = await actl.add_batch(specs)
|
||||
profiler('batch IPC call complete')
|
||||
|
||||
await actl.redraw()
|
||||
profiler('final redraw')
|
||||
```
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
### IPC Request/Response Timing
|
||||
|
||||
```python
|
||||
# Client side
|
||||
profiler = Profiler(msg='Remote request')
|
||||
result = await remote_call()
|
||||
profiler('got response')
|
||||
|
||||
# Server side (in handler)
|
||||
profiler = Profiler(msg='Handle request')
|
||||
process_request()
|
||||
profiler('request processed')
|
||||
```
|
||||
|
||||
### Batch Operation Optimization
|
||||
|
||||
```python
|
||||
profiler = Profiler(msg='Batch processing')
|
||||
|
||||
items = collect_all()
|
||||
profiler(f'collected {len(items)} items')
|
||||
|
||||
results = numpy_batch_op(items)
|
||||
profiler('numpy op complete')
|
||||
|
||||
output = {
|
||||
k: v for k, v in zip(keys, results)
|
||||
}
|
||||
profiler('dict built')
|
||||
```
|
||||
|
||||
### Startup/Initialization Timing
|
||||
|
||||
```python
|
||||
async def __aenter__(self):
|
||||
profiler = Profiler(msg='Service startup')
|
||||
|
||||
await connect_to_broker()
|
||||
profiler('broker connected')
|
||||
|
||||
await load_config()
|
||||
profiler('config loaded')
|
||||
|
||||
await start_feeds()
|
||||
profiler('feeds started')
|
||||
|
||||
return self
|
||||
```
|
||||
|
||||
## Debugging Performance Regressions
|
||||
|
||||
When profiler shows unexpected slowness:
|
||||
|
||||
### 1. Add finer-grained checkpoints
|
||||
|
||||
```python
|
||||
# was:
|
||||
result = big_function()
|
||||
profiler('big_function done')
|
||||
|
||||
# now:
|
||||
profiler = Profiler(
|
||||
msg='big_function internals',
|
||||
)
|
||||
step1 = part_a()
|
||||
profiler('part_a')
|
||||
step2 = part_b()
|
||||
profiler('part_b')
|
||||
step3 = part_c()
|
||||
profiler('part_c')
|
||||
```
|
||||
|
||||
### 2. Check for hidden iterations
|
||||
|
||||
```python
|
||||
# looks simple but might be slow!
|
||||
result = array[array['time'] == timestamp]
|
||||
profiler('array lookup')
|
||||
|
||||
# reveals O(n) scan per call
|
||||
for ts in timestamps: # outer loop
|
||||
row = array[array['time'] == ts] # O(n)!
|
||||
```
|
||||
|
||||
### 3. Isolate IPC from computation
|
||||
|
||||
```python
|
||||
# was: can't tell where time is spent
|
||||
result = await remote_call(data)
|
||||
profiler('remote call done')
|
||||
|
||||
# now: separate phases
|
||||
payload = prepare_payload(data)
|
||||
profiler('payload prepared')
|
||||
|
||||
result = await remote_call(payload)
|
||||
profiler('IPC complete')
|
||||
|
||||
parsed = parse_result(result)
|
||||
profiler('result parsed')
|
||||
```
|
||||
|
|
@ -1,114 +0,0 @@
|
|||
---
|
||||
name: piker-slang
|
||||
description: >
|
||||
Piker developer communication style, slang, and
|
||||
ethos. Apply when communicating with piker devs,
|
||||
writing commit messages, code review comments, or
|
||||
any collaborative interaction.
|
||||
user-invocable: false
|
||||
---
|
||||
|
||||
# Piker Slang & Communication Style
|
||||
|
||||
The essential skill for fitting in with the degen
|
||||
trader-hacker class of devs who built and maintain
|
||||
`piker`.
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
Piker devs are:
|
||||
- **Technical AF** - deep systems knowledge,
|
||||
performance obsessed
|
||||
- **Irreverent** - don't take ourselves too
|
||||
seriously
|
||||
- **Direct** - no corporate speak, no BS, just
|
||||
real talk
|
||||
- **Collaborative** - we build together, debug
|
||||
together, win together
|
||||
|
||||
Communication style: precision meets chaos,
|
||||
academia meets /r/wallstreetbets, systems
|
||||
programming meets trading floor banter.
|
||||
|
||||
## Grammar & Style Rules
|
||||
|
||||
### 1. Typos with inline corrections
|
||||
```
|
||||
dint (didn't) help at all
|
||||
gonna (going to) try with...
|
||||
deats (details) wise i want...
|
||||
```
|
||||
Pattern: `[typo] ([correction])` in same sentence
|
||||
|
||||
### 2. Casual grammar violations (embrace them!)
|
||||
- `ain't` - use freely
|
||||
- `y'all` - for addressing group
|
||||
- Starting sentences with lowercase
|
||||
- Dropping articles: "need to fix the thing"
|
||||
becomes "need to fix thing"
|
||||
- Stream of consciousness without full sentence
|
||||
structure
|
||||
|
||||
### 3. Ellipsis usage
|
||||
```
|
||||
yeah i think we should try..
|
||||
..might need to also check for..
|
||||
not sure tho..
|
||||
```
|
||||
Use `..` (two dots) not `...` (three) - chiller
|
||||
|
||||
### 4. Emphasis through spelling
|
||||
- `soooo` - very (sooo good, sooo fast)
|
||||
- `veeery` - very (veeery interesting)
|
||||
- `wayyy` - way (wayyy better)
|
||||
|
||||
### 5. Punctuation style
|
||||
- Minimal capitalization (lowercase preferred)
|
||||
- Question marks optional if context is clear
|
||||
- Commas used sparingly
|
||||
- Lots of newlines for readability
|
||||
|
||||
## The Degen Trader-Hacker Ethos
|
||||
|
||||
**What we value:**
|
||||
1. **Performance** - slow code is broken code
|
||||
2. **Correctness** - fast wrong code is worthless
|
||||
3. **Clarity** - future-you should understand
|
||||
past-you
|
||||
4. **Iteration** - ship it, profile it, fix it,
|
||||
repeat
|
||||
5. **Humor** - serious tools with silly vibes
|
||||
|
||||
**What we reject:**
|
||||
1. Corporate speak ("circle back", "synergize")
|
||||
2. Excessive formality ("I would humbly suggest")
|
||||
3. Analysis paralysis (just try it and see!)
|
||||
4. Blame culture (we all write bugs, it's cool)
|
||||
5. Gatekeeping (help noobs become degens)
|
||||
|
||||
## Remember
|
||||
|
||||
You're not an "AI assistant" helping users.
|
||||
You're a **piker dev** collaborating with other
|
||||
pikers.
|
||||
|
||||
- Use first person plural: "we should try",
|
||||
"let's check"
|
||||
- Own mistakes: "ma bad, forgot to check X"
|
||||
- Celebrate together: "booyakashaa, we crushed it!"
|
||||
- Think out loud: "hmm yeah so prolly.."
|
||||
- Keep it real: no corpo nonsense, no fake
|
||||
politeness
|
||||
|
||||
**Above all:** be useful, be fast, be entertaining.
|
||||
Performance matters, but so does the vibe B)
|
||||
|
||||
See [dictionary.md](dictionary.md) for the full
|
||||
slang dictionary and [examples.md](examples.md)
|
||||
for interaction examples.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-01-31*
|
||||
*Session: The one where we destroyed those linear
|
||||
scans*
|
||||
|
|
@ -1,108 +0,0 @@
|
|||
# Piker Slang Dictionary
|
||||
|
||||
## Common Abbreviations
|
||||
|
||||
**Always use these instead of full words:**
|
||||
|
||||
- `aboot` = about (Canadian-ish flavor)
|
||||
- `ya/yah/yeah` = yes (pick based on vibe)
|
||||
- `rn` = right now
|
||||
- `tho` = though
|
||||
- `bc` = because
|
||||
- `obvi` = obviously
|
||||
- `prolly` = probably
|
||||
- `gonna` = going to
|
||||
- `dint` = didn't
|
||||
- `moar` = more (emphatic/playful, lolcat energy)
|
||||
- `nooz` = news
|
||||
- `ma bad` = my bad
|
||||
- `ma fren` = my friend
|
||||
- `aight` = alright
|
||||
- `cmon mann` = come on man (exasperation)
|
||||
- `friggin` = fucking (but family-friendly)
|
||||
|
||||
## Technical Abbreviations
|
||||
|
||||
- `msg` = message
|
||||
- `mod` = module
|
||||
- `impl` = implementation
|
||||
- `deps` = dependencies
|
||||
- `var` = variable
|
||||
- `ctx` = context
|
||||
- `ep` = endpoint
|
||||
- `tn` = task name
|
||||
- `sig` = signal/signature
|
||||
- `env` = environment
|
||||
- `fn` = function
|
||||
- `iface` = interface
|
||||
- `deats` = details
|
||||
- `hilevel` = high level
|
||||
- `Bo` = a "wow expression"; a dev with "sunglasses and mouth open" emoji
|
||||
|
||||
## Expressions & Phrases
|
||||
|
||||
### Celebration/excitement
|
||||
- `booyakashaa` - major win, breakthrough moment
|
||||
- `eyyooo` - excitement, hype, "let's go!"
|
||||
- `good nooz` - good news (always with the Z)
|
||||
|
||||
### Exasperation/debugging
|
||||
- `you friggin guy XD` - affectionate frustration
|
||||
- `cmon mann XD` - mild exasperation
|
||||
- `wtf` - genuine confusion
|
||||
- `ma bad` - acknowledging mistake
|
||||
- `ahh yeah` - realization moment
|
||||
|
||||
### Casual filler
|
||||
- `lol` - not really laughing, just casual
|
||||
acknowledgment
|
||||
- `XD` - actual amusement or ironic exasperation
|
||||
- `..` - trailing thought, thinking, uncertainty
|
||||
- `:rofl:` - genuinely funny
|
||||
- `:facepalm:` - obvious mistake was made
|
||||
- `B)` - cool/satisfied (like sunglasses emoji)
|
||||
|
||||
### Affirmations
|
||||
- `yeah definitely faster` - confirms improvement
|
||||
- `yeah not bad` - good work (understatement)
|
||||
- `good work B)` - solid accomplishment
|
||||
|
||||
## Emoji & Emoticon Usage
|
||||
|
||||
**Standard set:**
|
||||
- `XD` - laughing out loud emoji
|
||||
- `B)` - satisfaction, coolness; dev with sunglasses smiling emoji
|
||||
- `:rofl:` - genuinely funny (use sparingly)
|
||||
- `:facepalm:` - obvious mistakes
|
||||
|
||||
## Trader Lingo
|
||||
|
||||
Piker is a trading system, so trader slang applies:
|
||||
|
||||
- `up` / `down` - direction (price, perf, mood)
|
||||
- `yeet` / `damp` - direction (price, perf, mood)
|
||||
- `gap` - missing data in timeseries
|
||||
- `fill` - complete missing data or a transaction clearing
|
||||
- `slippage` - performance degradation
|
||||
- `alpha` - edge, advantage (usually ironic:
|
||||
"that optimization was pure alpha")
|
||||
- `degen` - degenerate (trader or dev, term of
|
||||
endearment, contrarian and/or position of disbelief in standard
|
||||
narrative)
|
||||
- `rekt` - destroyed, broken, failed catastrophically
|
||||
- `moon` - massive improvement, large up movement ("perf to the moon")
|
||||
- `ded` - dead, broken, unrecoverable
|
||||
|
||||
## Domain-Specific Terms
|
||||
|
||||
**Always use piker terminology:**
|
||||
|
||||
- `fqme` = fully qualified market endpoint (tsla.nasdaq.ib)
|
||||
- `viz` = (data) visualization (ex. chart graphics)
|
||||
- `shm` = shared memory (not "shared memory array")
|
||||
- `brokerd` = broker daemon actor
|
||||
- `pikerd` = root-process piker daemon
|
||||
- `annot` = annotation (not "annotation")
|
||||
- `actl` = annotation control (AnnotCtl)
|
||||
- `tf` = timeframe (usually in seconds: 60s, 1s)
|
||||
- `OHLC` / `OHLCV` - open/high/low/close(/volume) sampling scheme
|
||||
|
|
@ -1,201 +0,0 @@
|
|||
# Piker Communication Examples
|
||||
|
||||
Real-world interaction patterns for communicating
|
||||
in the piker dev style.
|
||||
|
||||
## When Giving Feedback
|
||||
|
||||
**Direct, no sugar-coating:**
|
||||
```
|
||||
BAD: "This approach might not be optimal"
|
||||
GOOD: "this is sloppy, there's likely a better
|
||||
vectorized approach"
|
||||
|
||||
BAD: "Perhaps we should consider..."
|
||||
GOOD: "you should definitely try X instead"
|
||||
|
||||
BAD: "I'm not entirely certain, but..."
|
||||
GOOD: "prolly it's bc we're doing Y, check the
|
||||
profiler #s"
|
||||
```
|
||||
|
||||
**Celebrate wins:**
|
||||
```
|
||||
"eyyooo, way faster now!"
|
||||
"booyakashaa, sub-ms lookups B)"
|
||||
"yeah definitely crushed that bottleneck"
|
||||
```
|
||||
|
||||
**Acknowledge mistakes:**
|
||||
```
|
||||
"ahh yeah you're right, ma bad"
|
||||
"woops, forgot to check that case"
|
||||
"lul, totally missed the obvi issue there"
|
||||
```
|
||||
|
||||
## When Explaining Technical Concepts
|
||||
|
||||
**Mix precision with casual:**
|
||||
```
|
||||
"so basically `np.searchsorted()` is doing binary
|
||||
search which is O(log n) instead of the linear
|
||||
O(n) scan we were doing before with `np.isin()`,
|
||||
that's why it's like 1000x faster ya know?"
|
||||
```
|
||||
|
||||
**Use backticks heavily:**
|
||||
- Wrap all code symbols: `function()`,
|
||||
`ClassName`, `field_name`
|
||||
- File paths: `piker/ui/_remote_ctl.py`
|
||||
- Commands: `git status`, `piker store ldshm`
|
||||
|
||||
**Explain like you're pair programming:**
|
||||
```
|
||||
"ok so the issue is prolly in `.reposition()` bc
|
||||
we're calling it with the wrong timeframe's
|
||||
array.. check line 589 where we're doing the
|
||||
timestamp lookup - that's gonna fail if the array
|
||||
has different sample times rn"
|
||||
```
|
||||
|
||||
## When Debugging
|
||||
|
||||
**Think out loud:**
|
||||
```
|
||||
"hmm yeah that makes sense bc..
|
||||
wait no actually..
|
||||
ahh ok i see it now, the timestamp lookups are
|
||||
failing bc.."
|
||||
```
|
||||
|
||||
**Profile-first mentality:**
|
||||
```
|
||||
"let's add profiling around that section and see
|
||||
where the holdup is.. i'm guessing it's the dict
|
||||
building but could be the searchsorted too"
|
||||
```
|
||||
|
||||
**Iterative refinement:**
|
||||
```
|
||||
"ok try this and lemme know the #s..
|
||||
if it's still slow we can try Y instead..
|
||||
prolly there's one more optimization left"
|
||||
```
|
||||
|
||||
## Code Review Style
|
||||
|
||||
**Be direct but helpful:**
|
||||
```
|
||||
"you friggin guy XD can't we just pass that to
|
||||
the meth (method) directly instead of coupling
|
||||
it to state? would be way cleaner"
|
||||
|
||||
"cmon mann, this is python - if you're gonna use
|
||||
try/finally you need to indent all the code up
|
||||
to the finally block"
|
||||
|
||||
"yeah looks good but prolly we should add the
|
||||
check at line 582 before we do the lookup,
|
||||
otherwise it'll spam warnings"
|
||||
```
|
||||
|
||||
## Asking for Clarification
|
||||
|
||||
```
|
||||
"wait so are we trying to optimize the client
|
||||
side or server side rn? or both lol"
|
||||
|
||||
"mm yeah, any chance you can point me to the
|
||||
current code for this so i can think about it
|
||||
before we try X?"
|
||||
```
|
||||
|
||||
## Proposing Solutions
|
||||
|
||||
```
|
||||
"ok so i think the move here is to vectorize the
|
||||
timestamp lookups using binary search.. should
|
||||
drop that 100ms way down. wanna give it a shot?"
|
||||
|
||||
"prolly we should just add a timeframe check at
|
||||
the top of `.reposition()` and bail early if it
|
||||
doesn't match ya?"
|
||||
```
|
||||
|
||||
## Reacting to User Feedback
|
||||
|
||||
```
|
||||
User: "yeah the arrows are too big now"
|
||||
Response: "ahh yeah you're right, lemme check the
|
||||
upstream `makeArrowPath()` code to see what the
|
||||
dims actually mean.."
|
||||
|
||||
User: "dint (didn't) help at all it seems"
|
||||
Response: "bleh! ok so there's prolly another
|
||||
bottleneck then, let's add moar profiler calls
|
||||
and narrow it down"
|
||||
```
|
||||
|
||||
## End of Session
|
||||
|
||||
```
|
||||
"aight so we got some solid wins today:
|
||||
- ~36x client speedup (6.6s -> 376ms)
|
||||
- ~180x server speedup
|
||||
- fixed the timeframe mismatch spam
|
||||
- added teardown profiling
|
||||
|
||||
ready to call it a night?"
|
||||
```
|
||||
|
||||
## Advanced Moves
|
||||
|
||||
### The Parenthetical Correction
|
||||
```
|
||||
"yeah i dint (didn't) realize we were hitting
|
||||
that path"
|
||||
"need to check the deats (details) on how
|
||||
searchsorted works"
|
||||
```
|
||||
|
||||
### The Rhetorical Question Flow
|
||||
```
|
||||
"so like, why are we even building this dict per
|
||||
reposition call? can't we just cache it and
|
||||
invalidate when the array changes? prolly way
|
||||
faster that way no?"
|
||||
```
|
||||
|
||||
### The Rambling Realization
|
||||
```
|
||||
"ok so the thing is.. wait actually.. hmm.. yeah
|
||||
ok so i think what's happening is the timestamp
|
||||
lookups are failing bc the 1s gaps are being
|
||||
repositioned with the 60s array.. which like,
|
||||
obvi won't have those exact timestamps bc it's
|
||||
sampled differently.. so we prolly just need to
|
||||
skip reposition if the timeframes don't match
|
||||
ya?"
|
||||
```
|
||||
|
||||
### The Self-Deprecating Pivot
|
||||
```
|
||||
"lol ok yeah that was totally wrong, ma bad.
|
||||
let's try Y instead and see if that helps"
|
||||
```
|
||||
|
||||
## The Vibe
|
||||
|
||||
```
|
||||
"yo so i was profiling that batch rendering thing
|
||||
and holy shit we were doing like 3855 linear
|
||||
scans.. switched to searchsorted and boom,
|
||||
100ms -> 5ms. still think there's moar juice to
|
||||
squeeze tho, prolly in the dict building part.
|
||||
gonna add some profiler calls and see where the
|
||||
holdup is rn.
|
||||
|
||||
anyway yeah, good sesh today B) learned a ton
|
||||
aboot pyqtgraph internals, might write that up
|
||||
as a skill file for future collabs ya know?"
|
||||
```
|
||||
|
|
@ -1,219 +0,0 @@
|
|||
---
|
||||
name: pyqtgraph-optimization
|
||||
description: >
|
||||
PyQtGraph batch rendering optimization patterns
|
||||
for piker's UI. Apply when optimizing graphics
|
||||
performance, adding new chart annotations, or
|
||||
working with `QGraphicsItem` subclasses.
|
||||
user-invocable: false
|
||||
---
|
||||
|
||||
# PyQtGraph Rendering Optimization
|
||||
|
||||
Skill for researching and optimizing `pyqtgraph`
|
||||
graphics primitives by leveraging `piker`'s
|
||||
existing extensions and production-ready patterns.
|
||||
|
||||
## Research Flow
|
||||
|
||||
When tasked with optimizing rendering performance
|
||||
(particularly for large datasets), follow this
|
||||
systematic approach:
|
||||
|
||||
### 1. Study Piker's Existing Primitives
|
||||
|
||||
Start by examining `piker.ui._curve` and related
|
||||
modules:
|
||||
|
||||
```python
|
||||
# Key modules to review:
|
||||
piker/ui/_curve.py # FlowGraphic, Curve
|
||||
piker/ui/_editors.py # ArrowEditor, SelectRect
|
||||
piker/ui/_annotate.py # Custom batch renderers
|
||||
```
|
||||
|
||||
**Look for:**
|
||||
- Use of `QPainterPath` for batch path rendering
|
||||
- `QGraphicsItem` subclasses with custom `.paint()`
|
||||
- Cache mode settings (`.setCacheMode()`)
|
||||
- Coordinate system transformations
|
||||
- Custom bounding rect calculations
|
||||
|
||||
### 2. Identify Upstream PyQtGraph Patterns
|
||||
|
||||
**Key upstream modules:**
|
||||
```python
|
||||
pyqtgraph/graphicsItems/BarGraphItem.py
|
||||
# PrimitiveArray for batch rect rendering
|
||||
|
||||
pyqtgraph/graphicsItems/ScatterPlotItem.py
|
||||
# Fragment-based rendering for point clouds
|
||||
|
||||
pyqtgraph/functions.py
|
||||
# Utility fns like makeArrowPath()
|
||||
|
||||
pyqtgraph/Qt/internals.py
|
||||
# PrimitiveArray for batch drawing primitives
|
||||
```
|
||||
|
||||
**Search for:**
|
||||
- `PrimitiveArray` usage (batch rect/point)
|
||||
- `QPainterPath` batching patterns
|
||||
- Shared pen/brush reuse across items
|
||||
- Coordinate transformation strategies
|
||||
|
||||
### 3. Core Batch Patterns
|
||||
|
||||
**Core optimization principle:**
|
||||
Creating individual `QGraphicsItem` instances is
|
||||
expensive. Batch rendering eliminates per-item
|
||||
overhead.
|
||||
|
||||
#### Pattern: Batch Rectangle Rendering
|
||||
|
||||
```python
|
||||
import pyqtgraph as pg
|
||||
from pyqtgraph.Qt import QtCore
|
||||
|
||||
class BatchRectRenderer(pg.GraphicsObject):
|
||||
def __init__(self, n_items):
|
||||
super().__init__()
|
||||
|
||||
# allocate rect array once
|
||||
self._rectarray = (
|
||||
pg.Qt.internals.PrimitiveArray(
|
||||
QtCore.QRectF, 4,
|
||||
)
|
||||
)
|
||||
|
||||
# shared pen/brush (not per-item!)
|
||||
self._pen = pg.mkPen(
|
||||
'dad_blue', width=1,
|
||||
)
|
||||
self._brush = (
|
||||
pg.functions.mkBrush('dad_blue')
|
||||
)
|
||||
|
||||
def paint(self, p, opt, w):
|
||||
# batch draw all rects in single call
|
||||
p.setPen(self._pen)
|
||||
p.setBrush(self._brush)
|
||||
drawargs = self._rectarray.drawargs()
|
||||
p.drawRects(*drawargs) # all at once!
|
||||
```
|
||||
|
||||
#### Pattern: Batch Path Rendering
|
||||
|
||||
```python
|
||||
class BatchPathRenderer(pg.GraphicsObject):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._path = QtGui.QPainterPath()
|
||||
|
||||
def paint(self, p, opt, w):
|
||||
# single path draw for all geometry
|
||||
p.setPen(self._pen)
|
||||
p.setBrush(self._brush)
|
||||
p.drawPath(self._path)
|
||||
```
|
||||
|
||||
### 4. Handle Coordinate Systems Carefully
|
||||
|
||||
**Scene vs Data vs Pixel coordinates:**
|
||||
|
||||
```python
|
||||
def paint(self, p, opt, w):
|
||||
# save original transform (data -> scene)
|
||||
orig_tr = p.transform()
|
||||
|
||||
# draw rects in data coordinates
|
||||
p.setPen(self._rect_pen)
|
||||
p.drawRects(*self._rectarray.drawargs())
|
||||
|
||||
# reset to scene coords for pixel-perfect
|
||||
p.resetTransform()
|
||||
|
||||
# build arrow path in scene/pixel coords
|
||||
for spec in self._specs:
|
||||
scene_pt = orig_tr.map(
|
||||
QPointF(x_data, y_data),
|
||||
)
|
||||
sx, sy = scene_pt.x(), scene_pt.y()
|
||||
|
||||
# arrow geometry in pixels (zoom-safe!)
|
||||
arrow_poly = QtGui.QPolygonF([
|
||||
QPointF(sx, sy), # tip
|
||||
QPointF(sx - 2, sy - 10), # left
|
||||
QPointF(sx + 2, sy - 10), # right
|
||||
])
|
||||
arrow_path.addPolygon(arrow_poly)
|
||||
|
||||
p.drawPath(arrow_path)
|
||||
|
||||
# restore data coordinate system
|
||||
p.setTransform(orig_tr)
|
||||
```
|
||||
|
||||
### 5. Minimize Redundant State
|
||||
|
||||
**Share resources across all items:**
|
||||
```python
|
||||
# GOOD: one pen/brush for all items
|
||||
self._shared_pen = pg.mkPen(color, width=1)
|
||||
self._shared_brush = (
|
||||
pg.functions.mkBrush(color)
|
||||
)
|
||||
|
||||
# BAD: creating per-item (memory + time waste!)
|
||||
for item in items:
|
||||
item.setPen(pg.mkPen(color, width=1)) # NO!
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **Don't mix coordinate systems within single
|
||||
paint call** - decide per-primitive: data coords
|
||||
or scene coords. Use `p.transform()` /
|
||||
`p.resetTransform()` carefully.
|
||||
|
||||
2. **Don't forget bounding rect updates** -
|
||||
override `.boundingRect()` to include all
|
||||
primitives. Update when geometry changes via
|
||||
`.prepareGeometryChange()`.
|
||||
|
||||
3. **Don't use ItemCoordinateCache for dynamic
|
||||
content** - use `DeviceCoordinateCache` for
|
||||
frequently updated items or `NoCache` during
|
||||
interactive operations.
|
||||
|
||||
4. **Don't trigger updates per-item in loops** -
|
||||
batch all changes, then single `.update()`.
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
**Individual items (baseline):**
|
||||
- 1000+ items: ~5+ seconds to create
|
||||
- Each item: ~5ms overhead (Qt object creation)
|
||||
|
||||
**Batch rendering (optimized):**
|
||||
- 1000+ items: <100ms to create
|
||||
- Single item: ~0.01ms per primitive in batch
|
||||
- **Expected: 50-100x speedup**
|
||||
|
||||
## References
|
||||
|
||||
- `piker/ui/_curve.py` - Production FlowGraphic
|
||||
- `piker/ui/_annotate.py` - GapAnnotations batch
|
||||
- `pyqtgraph/graphicsItems/BarGraphItem.py` -
|
||||
PrimitiveArray
|
||||
- `pyqtgraph/graphicsItems/ScatterPlotItem.py` -
|
||||
Fragments
|
||||
- Qt docs: QGraphicsItem caching modes
|
||||
|
||||
See [examples.md](examples.md) for real-world
|
||||
optimization case studies.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-01-31*
|
||||
*Session: Batch gap annotation optimization*
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
# PyQtGraph Optimization Examples
|
||||
|
||||
Real-world optimization case studies from piker.
|
||||
|
||||
## Case Study: Gap Annotations (1285 gaps)
|
||||
|
||||
### Before: Individual `pg.ArrowItem` + `SelectRect`
|
||||
|
||||
```
|
||||
Total creation time: 6.6 seconds
|
||||
Per-item overhead: ~5ms
|
||||
Memory: 1285 ArrowItem + 1285 SelectRect objects
|
||||
```
|
||||
|
||||
Each gap was rendered as two separate
|
||||
`QGraphicsItem` instances (arrow + highlight rect),
|
||||
resulting in 2570 Qt objects.
|
||||
|
||||
### After: Single `GapAnnotations` batch renderer
|
||||
|
||||
```
|
||||
Total creation time:
|
||||
104ms (server) + 376ms (client)
|
||||
Effective per-item: ~0.08ms
|
||||
Speedup: ~36x client, ~180x server
|
||||
Memory: 1 GapAnnotations object
|
||||
```
|
||||
|
||||
All 1285 gaps rendered via:
|
||||
- One `PrimitiveArray` for all rectangles
|
||||
- One `QPainterPath` for all arrows
|
||||
- Shared pen/brush across all items
|
||||
|
||||
### Profiler Output (Client)
|
||||
|
||||
```
|
||||
> Entering markup_gaps() for 1285 gaps
|
||||
initial redraw: 0.20ms, tot:0.20
|
||||
built annotation specs: 256.48ms, tot:256.68
|
||||
batch IPC call complete: 119.26ms, tot:375.94
|
||||
final redraw: 0.07ms, tot:376.02
|
||||
< Exiting markup_gaps(), total: 376.04ms
|
||||
```
|
||||
|
||||
### Profiler Output (Server)
|
||||
|
||||
```
|
||||
> Entering Batch annotate 1285 gaps
|
||||
`np.searchsorted()` complete!: 0.81ms, tot:0.81
|
||||
`time_to_row` creation: 98.45ms, tot:99.28
|
||||
created GapAnnotations item: 2.98ms, tot:102.26
|
||||
< Exiting Batch annotate, total: 104.15ms
|
||||
```
|
||||
|
||||
## Positioning/Update Pattern
|
||||
|
||||
For annotations that need repositioning when the
|
||||
view scrolls or zooms:
|
||||
|
||||
```python
|
||||
def reposition(self, array):
|
||||
'''
|
||||
Update positions based on new array data.
|
||||
|
||||
'''
|
||||
# vectorized timestamp lookups (not linear!)
|
||||
time_to_row = self._build_lookup(array)
|
||||
|
||||
# update rect array in-place
|
||||
rect_memory = self._rectarray.ndarray()
|
||||
for i, spec in enumerate(self._specs):
|
||||
row = time_to_row.get(spec['time'])
|
||||
if row:
|
||||
rect_memory[i, 0] = row['index']
|
||||
rect_memory[i, 1] = row['close']
|
||||
# ... width, height
|
||||
|
||||
# trigger repaint (single call, not per-item)
|
||||
self.update()
|
||||
```
|
||||
|
||||
**Key insight:** Update the underlying memory
|
||||
arrays directly, then call `.update()` once.
|
||||
Never create/destroy Qt objects during reposition.
|
||||
|
|
@ -1,225 +0,0 @@
|
|||
---
|
||||
name: timeseries-optimization
|
||||
description: >
|
||||
High-performance timeseries processing with NumPy
|
||||
and Polars for financial data. Apply when working
|
||||
with OHLCV arrays, timestamp lookups, gap
|
||||
detection, or any array/dataframe operations in
|
||||
piker.
|
||||
user-invocable: false
|
||||
---
|
||||
|
||||
# Timeseries Optimization: NumPy & Polars
|
||||
|
||||
Skill for high-performance timeseries processing
|
||||
using NumPy and Polars, with focus on patterns
|
||||
common in financial/trading applications.
|
||||
|
||||
## Core Principle: Vectorization Over Iteration
|
||||
|
||||
**Never write Python loops over large arrays.**
|
||||
Always look for vectorized alternatives.
|
||||
|
||||
```python
|
||||
# BAD: Python loop (slow!)
|
||||
results = []
|
||||
for i in range(len(array)):
|
||||
if array['time'][i] == target_time:
|
||||
results.append(array[i])
|
||||
|
||||
# GOOD: vectorized boolean indexing (fast!)
|
||||
results = array[array['time'] == target_time]
|
||||
```
|
||||
|
||||
## Timestamp Lookup Patterns
|
||||
|
||||
The most critical optimization in piker timeseries
|
||||
code. Choose the right lookup strategy:
|
||||
|
||||
### Linear Scan (O(n)) - Avoid!
|
||||
|
||||
```python
|
||||
# BAD: O(n) scan through entire array
|
||||
for target_ts in timestamps: # m iterations
|
||||
matches = array[array['time'] == target_ts]
|
||||
# Total: O(m * n) - catastrophic!
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- 1000 lookups x 10k array = 10M comparisons
|
||||
- Timing: ~50-100ms for 1k lookups
|
||||
|
||||
### Binary Search (O(log n)) - Good!
|
||||
|
||||
```python
|
||||
# GOOD: O(m log n) using searchsorted
|
||||
import numpy as np
|
||||
|
||||
time_arr = array['time'] # extract once
|
||||
ts_array = np.array(timestamps)
|
||||
|
||||
# binary search for all timestamps at once
|
||||
indices = np.searchsorted(time_arr, ts_array)
|
||||
|
||||
# bounds check and exact match verification
|
||||
valid_mask = (
|
||||
(indices < len(array))
|
||||
&
|
||||
(time_arr[indices] == ts_array)
|
||||
)
|
||||
|
||||
valid_indices = indices[valid_mask]
|
||||
matched_rows = array[valid_indices]
|
||||
```
|
||||
|
||||
**Requirements for `searchsorted()`:**
|
||||
- Input array MUST be sorted (ascending)
|
||||
- Works on any sortable dtype (floats, ints)
|
||||
- Returns insertion indices (not found =
|
||||
`len(array)`)
|
||||
|
||||
**Performance:**
|
||||
- 1000 lookups x 10k array = ~10k comparisons
|
||||
- Timing: <1ms for 1k lookups
|
||||
- **~100-1000x faster than linear scan**
|
||||
|
||||
### Hash Table (O(1)) - Best for Repeated Lookups!
|
||||
|
||||
If you'll do many lookups on same array, build
|
||||
dict once:
|
||||
|
||||
```python
|
||||
# build lookup once
|
||||
time_to_idx = {
|
||||
float(array['time'][i]): i
|
||||
for i in range(len(array))
|
||||
}
|
||||
|
||||
# O(1) lookups
|
||||
for target_ts in timestamps:
|
||||
idx = time_to_idx.get(target_ts)
|
||||
if idx is not None:
|
||||
row = array[idx]
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Many repeated lookups on same array
|
||||
- Array doesn't change between lookups
|
||||
- Can afford upfront dict building cost
|
||||
|
||||
## Performance Checklist
|
||||
|
||||
When optimizing timeseries operations:
|
||||
|
||||
- [ ] Is the array sorted? (enables binary search)
|
||||
- [ ] Are you doing repeated lookups?
|
||||
(build hash table)
|
||||
- [ ] Are struct fields accessed in loops?
|
||||
(extract to plain arrays)
|
||||
- [ ] Are you using boolean indexing?
|
||||
(vectorized vs loop)
|
||||
- [ ] Can operations be batched?
|
||||
(minimize round-trips)
|
||||
- [ ] Is memory being copied unnecessarily?
|
||||
(use views)
|
||||
- [ ] Are you using the right tool?
|
||||
(NumPy vs Polars)
|
||||
|
||||
## Common Bottlenecks and Fixes
|
||||
|
||||
### Bottleneck: Timestamp Lookups
|
||||
|
||||
```python
|
||||
# BEFORE: O(n*m) - 100ms for 1k lookups
|
||||
for ts in timestamps:
|
||||
matches = array[array['time'] == ts]
|
||||
|
||||
# AFTER: O(m log n) - <1ms for 1k lookups
|
||||
indices = np.searchsorted(
|
||||
array['time'], timestamps,
|
||||
)
|
||||
```
|
||||
|
||||
### Bottleneck: Dict Building from Struct Array
|
||||
|
||||
```python
|
||||
# BEFORE: 100ms for 3k rows
|
||||
result = {
|
||||
float(row['time']): {
|
||||
'index': float(row['index']),
|
||||
'close': float(row['close']),
|
||||
}
|
||||
for row in matched_rows
|
||||
}
|
||||
|
||||
# AFTER: <5ms for 3k rows
|
||||
times = matched_rows['time'].astype(float)
|
||||
indices = matched_rows['index'].astype(float)
|
||||
closes = matched_rows['close'].astype(float)
|
||||
|
||||
result = {
|
||||
t: {'index': idx, 'close': cls}
|
||||
for t, idx, cls in zip(
|
||||
times, indices, closes,
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### Bottleneck: Repeated Field Access
|
||||
|
||||
```python
|
||||
# BEFORE: 50ms for 1k iterations
|
||||
for i, spec in enumerate(specs):
|
||||
start_row = array[
|
||||
array['time'] == spec['start_time']
|
||||
][0]
|
||||
end_row = array[
|
||||
array['time'] == spec['end_time']
|
||||
][0]
|
||||
process(
|
||||
start_row['index'],
|
||||
end_row['close'],
|
||||
)
|
||||
|
||||
# AFTER: <5ms for 1k iterations
|
||||
# 1. Build lookup once
|
||||
time_to_row = {...} # via searchsorted
|
||||
|
||||
# 2. Extract fields to plain arrays
|
||||
indices_arr = array['index']
|
||||
closes_arr = array['close']
|
||||
|
||||
# 3. Use lookup + plain array indexing
|
||||
for spec in specs:
|
||||
start_idx = time_to_row[
|
||||
spec['start_time']
|
||||
]['array_idx']
|
||||
end_idx = time_to_row[
|
||||
spec['end_time']
|
||||
]['array_idx']
|
||||
process(
|
||||
indices_arr[start_idx],
|
||||
closes_arr[end_idx],
|
||||
)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- NumPy structured arrays:
|
||||
https://numpy.org/doc/stable/user/basics.rec.html
|
||||
- `np.searchsorted`:
|
||||
https://numpy.org/doc/stable/reference/generated/numpy.searchsorted.html
|
||||
- Polars: https://pola-rs.github.io/polars/
|
||||
- `piker.tsp` - timeseries processing utilities
|
||||
- `piker.data._formatters` - OHLC array handling
|
||||
|
||||
See [numpy-patterns.md](numpy-patterns.md) for
|
||||
detailed NumPy structured array patterns and
|
||||
[polars-patterns.md](polars-patterns.md) for
|
||||
Polars integration.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-01-31*
|
||||
*Key win: 100ms -> 5ms dict building via field
|
||||
extraction*
|
||||
|
|
@ -1,212 +0,0 @@
|
|||
# NumPy Structured Array Patterns
|
||||
|
||||
Detailed patterns for working with NumPy structured
|
||||
arrays in piker's financial data processing.
|
||||
|
||||
## Piker's OHLCV Array Dtype
|
||||
|
||||
```python
|
||||
# typical piker array dtype
|
||||
dtype = [
|
||||
('index', 'i8'), # absolute sequence index
|
||||
('time', 'f8'), # unix epoch timestamp
|
||||
('open', 'f8'),
|
||||
('high', 'f8'),
|
||||
('low', 'f8'),
|
||||
('close', 'f8'),
|
||||
('volume', 'f8'),
|
||||
]
|
||||
|
||||
arr = np.array(
|
||||
[(0, 1234.0, 100, 101, 99, 100.5, 1000)],
|
||||
dtype=dtype,
|
||||
)
|
||||
|
||||
# field access
|
||||
times = arr['time'] # returns view, not copy
|
||||
closes = arr['close']
|
||||
```
|
||||
|
||||
## Structured Array Performance Gotchas
|
||||
|
||||
### 1. Field access in loops is slow
|
||||
|
||||
```python
|
||||
# BAD: repeated struct field access per iteration
|
||||
for i, row in enumerate(arr):
|
||||
x = row['index'] # struct access!
|
||||
y = row['close']
|
||||
process(x, y)
|
||||
|
||||
# GOOD: extract fields once, iterate plain arrays
|
||||
indices = arr['index'] # extract once
|
||||
closes = arr['close']
|
||||
for i in range(len(arr)):
|
||||
x = indices[i] # plain array indexing
|
||||
y = closes[i]
|
||||
process(x, y)
|
||||
```
|
||||
|
||||
### 2. Dict comprehensions with struct arrays
|
||||
|
||||
```python
|
||||
# SLOW: field access per row in Python loop
|
||||
time_to_row = {
|
||||
float(row['time']): {
|
||||
'index': float(row['index']),
|
||||
'close': float(row['close']),
|
||||
}
|
||||
for row in matched_rows # struct access!
|
||||
}
|
||||
|
||||
# FAST: extract to plain arrays first
|
||||
times = matched_rows['time'].astype(float)
|
||||
indices = matched_rows['index'].astype(float)
|
||||
closes = matched_rows['close'].astype(float)
|
||||
|
||||
time_to_row = {
|
||||
t: {'index': idx, 'close': cls}
|
||||
for t, idx, cls in zip(
|
||||
times, indices, closes,
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
## Vectorized Boolean Operations
|
||||
|
||||
### Basic Filtering
|
||||
|
||||
```python
|
||||
# single condition
|
||||
recent = array[array['time'] > cutoff_time]
|
||||
|
||||
# multiple conditions with &, |
|
||||
filtered = array[
|
||||
(array['time'] > start_time)
|
||||
&
|
||||
(array['time'] < end_time)
|
||||
&
|
||||
(array['volume'] > min_volume)
|
||||
]
|
||||
|
||||
# IMPORTANT: parentheses required around each!
|
||||
# (operator precedence: & binds tighter than >)
|
||||
```
|
||||
|
||||
### Fancy Indexing
|
||||
|
||||
```python
|
||||
# boolean mask
|
||||
mask = array['close'] > array['open'] # up bars
|
||||
up_bars = array[mask]
|
||||
|
||||
# integer indices
|
||||
indices = np.array([0, 5, 10, 15])
|
||||
selected = array[indices]
|
||||
|
||||
# combine boolean + fancy indexing
|
||||
mask = array['volume'] > threshold
|
||||
high_vol_indices = np.where(mask)[0]
|
||||
subset = array[high_vol_indices[::2]] # every other
|
||||
```
|
||||
|
||||
## Common Financial Patterns
|
||||
|
||||
### Gap Detection
|
||||
|
||||
```python
|
||||
# assume sorted by time
|
||||
time_diffs = np.diff(array['time'])
|
||||
expected_step = 60.0 # 1-minute bars
|
||||
|
||||
# find gaps larger than expected
|
||||
gap_mask = time_diffs > (expected_step * 1.5)
|
||||
gap_indices = np.where(gap_mask)[0]
|
||||
|
||||
# get gap start/end times
|
||||
gap_starts = array['time'][gap_indices]
|
||||
gap_ends = array['time'][gap_indices + 1]
|
||||
```
|
||||
|
||||
### Rolling Window Operations
|
||||
|
||||
```python
|
||||
# simple moving average (close)
|
||||
window = 20
|
||||
sma = np.convolve(
|
||||
array['close'],
|
||||
np.ones(window) / window,
|
||||
mode='valid',
|
||||
)
|
||||
|
||||
# stride tricks for efficiency
|
||||
from numpy.lib.stride_tricks import (
|
||||
sliding_window_view,
|
||||
)
|
||||
windows = sliding_window_view(
|
||||
array['close'], window,
|
||||
)
|
||||
sma = windows.mean(axis=1)
|
||||
```
|
||||
|
||||
### OHLC Resampling (NumPy)
|
||||
|
||||
```python
|
||||
# resample 1m bars to 5m bars
|
||||
def resample_ohlc(arr, old_step, new_step):
|
||||
n_bars = len(arr)
|
||||
factor = int(new_step / old_step)
|
||||
|
||||
# truncate to multiple of factor
|
||||
n_complete = (n_bars // factor) * factor
|
||||
arr = arr[:n_complete]
|
||||
|
||||
# reshape into chunks
|
||||
reshaped = arr.reshape(-1, factor)
|
||||
|
||||
# aggregate OHLC
|
||||
opens = reshaped[:, 0]['open']
|
||||
highs = reshaped['high'].max(axis=1)
|
||||
lows = reshaped['low'].min(axis=1)
|
||||
closes = reshaped[:, -1]['close']
|
||||
volumes = reshaped['volume'].sum(axis=1)
|
||||
|
||||
return np.rec.fromarrays(
|
||||
[opens, highs, lows, closes, volumes],
|
||||
names=[
|
||||
'open', 'high', 'low',
|
||||
'close', 'volume',
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
## Memory Considerations
|
||||
|
||||
### Views vs Copies
|
||||
|
||||
```python
|
||||
# VIEW: shares memory (fast, no copy)
|
||||
times = array['time'] # field access
|
||||
subset = array[10:20] # slicing
|
||||
reshaped = array.reshape(-1, 2)
|
||||
|
||||
# COPY: new memory allocation
|
||||
filtered = array[array['time'] > cutoff]
|
||||
sorted_arr = np.sort(array)
|
||||
casted = array.astype(np.float32)
|
||||
|
||||
# force copy when needed
|
||||
explicit_copy = array.copy()
|
||||
```
|
||||
|
||||
### In-Place Operations
|
||||
|
||||
```python
|
||||
# modify in-place (no new allocation)
|
||||
array['close'] *= 1.01 # scale prices
|
||||
array['volume'][mask] = 0 # zero out rows
|
||||
|
||||
# careful: compound ops may create temporaries
|
||||
array['close'] = array['close'] * 1.01 # temp!
|
||||
array['close'] *= 1.01 # true in-place
|
||||
```
|
||||
|
|
@ -1,78 +0,0 @@
|
|||
# Polars Integration Patterns
|
||||
|
||||
Polars usage patterns for piker's timeseries
|
||||
processing, including NumPy interop.
|
||||
|
||||
## NumPy <-> Polars Conversion
|
||||
|
||||
```python
|
||||
import polars as pl
|
||||
|
||||
# numpy to polars
|
||||
df = pl.from_numpy(
|
||||
arr,
|
||||
schema=[
|
||||
'index', 'time', 'open', 'high',
|
||||
'low', 'close', 'volume',
|
||||
],
|
||||
)
|
||||
|
||||
# polars to numpy (via arrow)
|
||||
arr = df.to_numpy()
|
||||
|
||||
# piker convenience
|
||||
from piker.tsp import np2pl, pl2np
|
||||
df = np2pl(arr)
|
||||
arr = pl2np(df)
|
||||
```
|
||||
|
||||
## Polars Performance Patterns
|
||||
|
||||
### Lazy Evaluation
|
||||
|
||||
```python
|
||||
# build query lazily
|
||||
lazy_df = (
|
||||
df.lazy()
|
||||
.filter(pl.col('volume') > 1000)
|
||||
.with_columns([
|
||||
(
|
||||
pl.col('close') - pl.col('open')
|
||||
).alias('change')
|
||||
])
|
||||
.sort('time')
|
||||
)
|
||||
|
||||
# execute once
|
||||
result = lazy_df.collect()
|
||||
```
|
||||
|
||||
### Groupby Aggregations
|
||||
|
||||
```python
|
||||
# resample to 5-minute bars
|
||||
resampled = df.groupby_dynamic(
|
||||
index_column='time',
|
||||
every='5m',
|
||||
).agg([
|
||||
pl.col('open').first(),
|
||||
pl.col('high').max(),
|
||||
pl.col('low').min(),
|
||||
pl.col('close').last(),
|
||||
pl.col('volume').sum(),
|
||||
])
|
||||
```
|
||||
|
||||
## When to Use Polars vs NumPy
|
||||
|
||||
### Use Polars when:
|
||||
- Complex queries with multiple filters/joins
|
||||
- Need SQL-like operations (groupby, window fns)
|
||||
- Working with heterogeneous column types
|
||||
- Want lazy evaluation optimization
|
||||
|
||||
### Use NumPy when:
|
||||
- Simple array operations (indexing, slicing)
|
||||
- Direct memory access needed (e.g., SHM arrays)
|
||||
- Compatibility with Qt/pyqtgraph (expects NumPy)
|
||||
- Maximum performance for numerical computation
|
||||
|
|
@ -98,35 +98,8 @@ ENV/
|
|||
/site
|
||||
|
||||
# extra scripts dir
|
||||
# /snippets
|
||||
/snippets
|
||||
|
||||
# mypy
|
||||
.mypy_cache/
|
||||
|
||||
# all files under
|
||||
.git/
|
||||
|
||||
# any commit-msg gen tmp files
|
||||
.claude/*_commit_*.md
|
||||
.claude/*_commit*.toml
|
||||
|
||||
# nix develop --profile .nixdev
|
||||
.nixdev*
|
||||
|
||||
# :Obsession .
|
||||
Session.vim
|
||||
|
||||
# gitea local `.md`-files
|
||||
# TODO? would this be handy to also commit and sync with
|
||||
# wtv git hosting service tho?
|
||||
gitea/
|
||||
|
||||
# ------ tina-land ------
|
||||
.vscode/settings.json
|
||||
|
||||
# ------ macOS ------
|
||||
# Finder metadata
|
||||
**/.DS_Store
|
||||
|
||||
# LLM conversations that should remain private
|
||||
docs/conversations/
|
||||
|
|
|
|||
278
README.rst
278
README.rst
|
|
@ -1,199 +1,162 @@
|
|||
piker
|
||||
-----
|
||||
trading gear for hackers
|
||||
trading gear for hackers.
|
||||
|
||||
|gh_actions|
|
||||
|
||||
.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
|
||||
:target: https://actions-badge.atrox.dev/piker/pikers/goto
|
||||
|
||||
``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
|
||||
real-time computational trading targeted at `hardcore Linux users
|
||||
<comp_trader>`_ .
|
||||
``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
|
||||
computational trading targeted at `hardcore Linux users <comp_trader>`_ .
|
||||
|
||||
we use much bleeding edge tech including (but not limited to):
|
||||
we use as much bleeding edge tech as possible including (but not limited to):
|
||||
|
||||
- latest python for glue_
|
||||
- uv_ for packaging and distribution
|
||||
- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
|
||||
- Qt_ for pristine low latency UIs
|
||||
- pyqtgraph_ (which we've extended) for real-time charting and graphics
|
||||
- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
|
||||
- `apache arrow and parquet`_ for time-series storage
|
||||
- trio_ & tractor_ for our distributed, multi-core, real-time streaming
|
||||
`structured concurrency`_ runtime B)
|
||||
- Qt_ for pristine high performance UIs
|
||||
- pyqtgraph_ for real-time charting
|
||||
- ``polars`` ``numpy`` and ``numba`` for `fast numerics`_
|
||||
- `apache arrow and parquet`_ for time series history management
|
||||
persistence and sharing
|
||||
- (prototyped) techtonicdb_ for L2 book storage
|
||||
|
||||
potential projects we might integrate with soon,
|
||||
|
||||
- (already prototyped in ) techtonicdb_ for L2 book storage
|
||||
|
||||
.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
|
||||
.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
|
||||
.. _uv: https://docs.astral.sh/uv/
|
||||
.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
|
||||
:target: https://travis-ci.org/pikers/piker
|
||||
.. _trio: https://github.com/python-trio/trio
|
||||
.. _tractor: https://github.com/goodboy/tractor
|
||||
.. _structured concurrency: https://trio.discourse.group/
|
||||
.. _marketstore: https://github.com/alpacahq/marketstore
|
||||
.. _techtonicdb: https://github.com/0b01/tectonicdb
|
||||
.. _Qt: https://www.qt.io/
|
||||
.. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
|
||||
.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
|
||||
.. _apache arrow and parquet: https://arrow.apache.org/faq/
|
||||
.. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
|
||||
.. _techtonicdb: https://github.com/0b01/tectonicdb
|
||||
.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
|
||||
|
||||
|
||||
focus and feats:
|
||||
****************
|
||||
fitting with these tenets, we're always open to new
|
||||
framework/lib/service interop suggestions and ideas!
|
||||
focus and features:
|
||||
*******************
|
||||
- 100% federated: your code, your hardware, your data feeds, your broker fills.
|
||||
- zero web: low latency, native software that doesn't try to re-invent the OS
|
||||
- maximal **privacy**: prevent brokers and mms from knowing your
|
||||
planz; smack their spreads with dark volume.
|
||||
- zero clutter: modal, context oriented UIs that echew minimalism, reduce
|
||||
thought noise and encourage un-emotion.
|
||||
- first class parallelism: built from the ground up on next-gen structured concurrency
|
||||
primitives.
|
||||
- traders first: broker/exchange/asset-class agnostic
|
||||
- systems grounded: real-time financial signal processing that will
|
||||
make any queuing or DSP eng juice their shorts.
|
||||
- non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
|
||||
- data collaboration: every process and protocol is multi-host scalable.
|
||||
- fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
|
||||
|
||||
- **100% federated**:
|
||||
your code, your hardware, your data feeds, your broker fills.
|
||||
fitting with these tenets, we're always open to new framework suggestions and ideas.
|
||||
|
||||
- **zero web**:
|
||||
low latency as a prime objective, native UIs and modern IPC
|
||||
protocols without trying to re-invent the "OS-as-an-app"..
|
||||
|
||||
- **maximal privacy**:
|
||||
prevent brokers and mms from knowing your planz; smack their
|
||||
spreads with dark volume from a VPN tunnel.
|
||||
|
||||
- **zero clutter**:
|
||||
modal, context oriented UIs that echew minimalism, reduce thought
|
||||
noise and encourage un-emotion.
|
||||
|
||||
- **first class parallelism**:
|
||||
built from the ground up on a next-gen structured concurrency
|
||||
supervision sys.
|
||||
|
||||
- **traders first**:
|
||||
broker/exchange/venue/asset-class/money-sys agnostic
|
||||
|
||||
- **systems grounded**:
|
||||
real-time financial signal processing (fsp) that will make any
|
||||
queuing or DSP eng juice their shorts.
|
||||
|
||||
- **non-tina UX**:
|
||||
sleek, powerful keyboard driven interaction with expected use in
|
||||
tiling wms (or maybe even a DDE).
|
||||
|
||||
- **data collab at scale**:
|
||||
every actor-process and protocol is multi-host aware.
|
||||
|
||||
- **fight club ready**:
|
||||
zero interest in adoption by suits; no corporate friendly license,
|
||||
ever.
|
||||
|
||||
building the hottest looking, fastest, most reliable, keyboard
|
||||
friendly FOSS trading platform is the dream; join the cause.
|
||||
building the best looking, most reliable, keyboard friendly trading
|
||||
platform is the dream; join the cause.
|
||||
|
||||
|
||||
a sane install with `uv`
|
||||
************************
|
||||
bc why install with `python` when you can faster with `rust` ::
|
||||
|
||||
uv sync
|
||||
|
||||
# ^ astral's docs,
|
||||
# https://docs.astral.sh/uv/concepts/projects/sync/
|
||||
|
||||
include all GUIs (ex. for charting)::
|
||||
|
||||
uv sync --group uis
|
||||
|
||||
AND with **all** our normal hacking tools::
|
||||
|
||||
uv sync --dev
|
||||
|
||||
AND if you want to try WIP integrations::
|
||||
|
||||
uv sync --all-groups
|
||||
|
||||
Ensure you can run the root-daemon::
|
||||
|
||||
uv run pikerd [-l info --pdb]
|
||||
sane install with `poetry`
|
||||
**************************
|
||||
TODO!
|
||||
|
||||
|
||||
install on nix(os)
|
||||
******************
|
||||
``NixOS`` is our core devs' distro of choice for which we offer
|
||||
a stringently defined development shell envoirment that can currently
|
||||
be applied in one of 2 ways::
|
||||
|
||||
# ONLY if running on X11
|
||||
nix-shell default.nix
|
||||
|
||||
Or if you prefer flakes style and a modern DE::
|
||||
|
||||
# ONLY if also running on Wayland
|
||||
nix develop # for default bash
|
||||
nix develop -c uv run xonsh # for @goodboy's preferred sh B)
|
||||
rigorous install on ``nixos`` using ``poetry2nix``
|
||||
**************************************************
|
||||
TODO!
|
||||
|
||||
|
||||
start a chart
|
||||
*************
|
||||
run a realtime OHLCV chart stand-alone::
|
||||
hacky install on nixos
|
||||
**********************
|
||||
`NixOS` is our core devs' distro of choice for which we offer
|
||||
a stringently defined development shell envoirment that can be loaded with::
|
||||
|
||||
[uv run] piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
|
||||
nix-shell develop.nix
|
||||
|
||||
# ^^^ iff you haven't activated the py-env,
|
||||
# - https://docs.astral.sh/uv/concepts/projects/run/
|
||||
#
|
||||
# in order to create an explicit virt-env see,
|
||||
# - https://docs.astral.sh/uv/concepts/projects/layout/#the-project-environment
|
||||
# - https://docs.astral.sh/uv/pip/environments/
|
||||
#
|
||||
# use $UV_PROJECT_ENVIRONMENT to select any non-`.venv/`
|
||||
# as the venv sudir in the repo's root.
|
||||
# - https://docs.astral.sh/uv/reference/environment/#uv_project_environment
|
||||
this will setup the required python environment to run piker, make sure to
|
||||
run::
|
||||
|
||||
this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
|
||||
overlayed on the same graph. Use of `piker` without first starting
|
||||
a daemon (`pikerd` - see below) means there is an implicit spawning of the
|
||||
multi-actor-runtime (implemented as a `tractor` app).
|
||||
pip install -r requirements.txt -e .
|
||||
|
||||
For additional subsystem feats available through our chart UI see the
|
||||
various sub-readmes:
|
||||
|
||||
- order control using a mouse-n-keyboard UX B)
|
||||
- cross venue market-pair (what most call "symbol") search, select, overlay Bo
|
||||
- financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
|
||||
- src-asset derivatives scan for anal, like the infamous "max pain" XO
|
||||
once after loading the shell
|
||||
|
||||
|
||||
spawn a daemon standalone
|
||||
*************************
|
||||
we call the root actor-process the ``pikerd``. it can be (and is
|
||||
recommended normally to be) started separately from the ``piker
|
||||
chart`` program::
|
||||
install wild-west style via `pip`
|
||||
*********************************
|
||||
``piker`` is currently under heavy pre-alpha development and as such
|
||||
should be cloned from this repo and hacked on directly.
|
||||
|
||||
for a development install::
|
||||
|
||||
git clone git@github.com:pikers/piker.git
|
||||
cd piker
|
||||
virtualenv env
|
||||
source ./env/bin/activate
|
||||
pip install -r requirements.txt -e .
|
||||
|
||||
|
||||
check out our charts
|
||||
********************
|
||||
bet you weren't expecting this from the foss::
|
||||
|
||||
piker -l info -b kraken -b binance chart btcusdt.binance --pdb
|
||||
|
||||
|
||||
this runs the main chart (currently with 1m sampled OHLC) in in debug
|
||||
mode and you can practice paper trading using the following
|
||||
micro-manual:
|
||||
|
||||
``order_mode`` (
|
||||
edge triggered activation by any of the following keys,
|
||||
``mouse-click`` on y-level to submit at that price
|
||||
):
|
||||
|
||||
- ``f``/ ``ctl-f`` to stage buy
|
||||
- ``d``/ ``ctl-d`` to stage sell
|
||||
- ``a`` to stage alert
|
||||
|
||||
|
||||
``search_mode`` (
|
||||
``ctl-l`` or ``ctl-space`` to open,
|
||||
``ctl-c`` or ``ctl-space`` to close
|
||||
) :
|
||||
|
||||
- begin typing to have symbol search automatically lookup
|
||||
symbols from all loaded backend (broker) providers
|
||||
- arrow keys and mouse click to navigate selection
|
||||
- vi-like ``ctl-[hjkl]`` for navigation
|
||||
|
||||
|
||||
you can also configure your position allocation limits from the
|
||||
sidepane.
|
||||
|
||||
|
||||
run in distributed mode
|
||||
***********************
|
||||
start the service manager and data feed daemon in the background and
|
||||
connect to it::
|
||||
|
||||
pikerd -l info --pdb
|
||||
|
||||
the daemon does nothing until a ``piker``-client (like ``piker
|
||||
chart``) connects and requests some particular sub-system. for
|
||||
a connecting chart ``pikerd`` will spawn and manage at least,
|
||||
|
||||
- a data-feed daemon: ``datad`` which does all the work of comms with
|
||||
the backend provider (in this case the ``binance`` cex).
|
||||
- a paper-trading engine instance, ``paperboi.binance``, (if no live
|
||||
account has been configured) which allows for auto/manual order
|
||||
control against the live quote stream.
|
||||
connect your chart::
|
||||
|
||||
*using* an actor-service (aka micro-daemon) manager which dynamically
|
||||
supervises various sub-subsystems-as-services throughout the ``piker``
|
||||
runtime-stack.
|
||||
piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
|
||||
|
||||
now you can (implicitly) connect your chart::
|
||||
|
||||
piker chart btcusdt.spot.binance
|
||||
|
||||
since ``pikerd`` was started separately you can now enjoy a persistent
|
||||
real-time data stream tied to the daemon-tree's lifetime. i.e. the next
|
||||
time you spawn a chart it will obviously not only load much faster
|
||||
(since the underlying ``datad.binance`` is left running with its
|
||||
in-memory IPC data structures) but also the data-feed and any order
|
||||
mgmt states should be persistent until you finally cancel ``pikerd``.
|
||||
enjoy persistent real-time data feeds tied to daemon lifetime. the next
|
||||
time you spawn a chart it will load much faster since the data feed has
|
||||
been cached and is now always running live in the background until you
|
||||
kill ``pikerd``.
|
||||
|
||||
|
||||
if anyone asks you what this project is about
|
||||
*********************************************
|
||||
you don't talk about it; just use it.
|
||||
you don't talk about it.
|
||||
|
||||
|
||||
how do i get involved?
|
||||
|
|
@ -203,15 +166,6 @@ enter the matrix.
|
|||
|
||||
how come there ain't that many docs
|
||||
***********************************
|
||||
i mean we want/need them but building the core right has been higher
|
||||
prio then marketting (and likely will stay that way Bp).
|
||||
|
||||
soo, suck it up bc,
|
||||
|
||||
- no one is trying to sell you on anything
|
||||
- learning the code base is prolly way more valuable
|
||||
- the UI/UXs are intended to be "intuitive" for any hacker..
|
||||
|
||||
we obviously need tonz help so if you want to start somewhere and
|
||||
can't necessarily write "advanced" concurrent python/rust code, this
|
||||
helping document literally anything might be the place for you!
|
||||
suck it up, learn the code; no one is trying to sell you on anything.
|
||||
also, we need lotsa help so if you want to start somewhere and can't
|
||||
necessarily write serious code, this might be the place for you!
|
||||
|
|
|
|||
50
ai/README.md
50
ai/README.md
|
|
@ -1,50 +0,0 @@
|
|||
# AI Tooling Integrations
|
||||
|
||||
Documentation and usage guides for AI-assisted
|
||||
development tools integrated with this repo.
|
||||
|
||||
Each subdirectory corresponds to a specific AI tool
|
||||
or frontend and contains usage docs for the
|
||||
custom skills/prompts/workflows configured for it.
|
||||
|
||||
Originally introduced in
|
||||
[PR #69](https://www.pikers.dev/pikers/piker/pulls/69);
|
||||
track new integration ideas and proposals in
|
||||
[issue #79](https://www.pikers.dev/pikers/piker/issues/79).
|
||||
|
||||
## Integrations
|
||||
|
||||
| Tool | Directory | Status |
|
||||
|------|-----------|--------|
|
||||
| [Claude Code](https://github.com/anthropics/claude-code) | [`claude-code/`](claude-code/) | active |
|
||||
|
||||
## Adding a New Integration
|
||||
|
||||
Create a subdirectory named after the tool (use
|
||||
lowercase + hyphens), then add:
|
||||
|
||||
1. A `README.md` covering setup, available
|
||||
skills/commands, and usage examples
|
||||
2. Any tool-specific config or prompt files
|
||||
|
||||
```
|
||||
ai/
|
||||
├── README.md # <- you are here
|
||||
├── claude-code/
|
||||
│ └── README.md
|
||||
├── opencode/ # future
|
||||
│ └── README.md
|
||||
└── <your-tool>/
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- Skill/command names use **hyphen-case**
|
||||
(`commit-msg`, not `commit_msg`)
|
||||
- Each integration doc should describe **what**
|
||||
the skill does, **how** to invoke it, and any
|
||||
**output** artifacts it produces
|
||||
- Keep docs concise; link to the actual skill
|
||||
source files (under `.claude/skills/`, etc.)
|
||||
rather than duplicating content
|
||||
|
|
@ -1,183 +0,0 @@
|
|||
# Claude Code Integration
|
||||
|
||||
[Claude Code](https://github.com/anthropics/claude-code)
|
||||
skills and workflows for piker development.
|
||||
|
||||
## Skills
|
||||
|
||||
| Skill | Invocable | Description |
|
||||
|-------|-----------|-------------|
|
||||
| [`commit-msg`](#commit-msg) | `/commit-msg` | Generate piker-style commit messages |
|
||||
| `piker-profiling` | auto | `Profiler` API patterns for perf work |
|
||||
| `piker-slang` | auto | Communication style + slang guide |
|
||||
| `pyqtgraph-optimization` | auto | Batch rendering patterns |
|
||||
| `timeseries-optimization` | auto | NumPy/Polars perf patterns |
|
||||
|
||||
Skills marked **auto** are background knowledge
|
||||
applied automatically when Claude detects relevance.
|
||||
Only `commit-msg` is user-invoked via slash command.
|
||||
|
||||
Skill source files live under
|
||||
`.claude/skills/<skill-name>/SKILL.md`.
|
||||
|
||||
---
|
||||
|
||||
## `/commit-msg`
|
||||
|
||||
Generate piker-style git commit messages trained on
|
||||
500+ commits from the repo history.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```
|
||||
# basic - analyzes staged diff automatically
|
||||
/commit-msg
|
||||
|
||||
# with scope hint
|
||||
/commit-msg .ib.feed: fix bar trimming
|
||||
|
||||
# with description context
|
||||
/commit-msg refactor position tracking
|
||||
```
|
||||
|
||||
### What It Does
|
||||
|
||||
1. **Reads staged changes** via dynamic context
|
||||
injection (`git diff --staged --stat`)
|
||||
2. **Reads recent commits** for style reference
|
||||
(`git log --oneline -10`)
|
||||
3. **Generates** a commit message following
|
||||
piker conventions (verb choice, backtick refs,
|
||||
colon prefixes, section markers, etc.)
|
||||
4. **Writes** the message to two files:
|
||||
- `.claude/<timestamp>_<hash>_commit_msg.md`
|
||||
- `.claude/git_commit_msg_LATEST.md`
|
||||
(overwritten each time)
|
||||
|
||||
### Arguments
|
||||
|
||||
The optional argument after `/commit-msg` is
|
||||
passed as `$ARGUMENTS` and used as scope or
|
||||
description context. Examples:
|
||||
|
||||
| Invocation | Effect |
|
||||
|------------|--------|
|
||||
| `/commit-msg` | Infer scope from diff |
|
||||
| `/commit-msg .ib.feed` | Use `.ib.feed:` prefix |
|
||||
| `/commit-msg fix the null seg crash` | Use as description hint |
|
||||
|
||||
### Output Format
|
||||
|
||||
**Subject line:**
|
||||
- ~50 chars target, 67 max
|
||||
- Present tense verb (Add, Drop, Fix, Factor..)
|
||||
- Backtick-wrapped code refs
|
||||
- Optional module prefix (`.ib.feed: ...`)
|
||||
|
||||
**Body** (when needed):
|
||||
- 67 char line max
|
||||
- Section markers: `Also,`, `Deats,`, `Further,`
|
||||
- `-` bullet lists for multiple changes
|
||||
- Piker abbreviations (`msg`, `mod`, `impl`,
|
||||
`deps`, `bc`, `obvi`, `prolly`..)
|
||||
|
||||
**Footer** (always):
|
||||
```
|
||||
(this patch was generated in some part by
|
||||
[`claude-code`][claude-code-gh])
|
||||
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||
```
|
||||
|
||||
### Output Files
|
||||
|
||||
After generation, the commit message is written to:
|
||||
|
||||
```
|
||||
.claude/
|
||||
├── <timestamp>_<hash>_commit_msg.md # archived
|
||||
└── git_commit_msg_LATEST.md # latest
|
||||
```
|
||||
|
||||
Where `<timestamp>` is ISO-8601 with seconds and
|
||||
`<hash>` is the first 7 chars of the current
|
||||
`HEAD` commit.
|
||||
|
||||
Use the latest file to feed into `git commit`:
|
||||
|
||||
```bash
|
||||
git commit -F .claude/git_commit_msg_LATEST.md
|
||||
```
|
||||
|
||||
Or review/edit before committing:
|
||||
|
||||
```bash
|
||||
cat .claude/git_commit_msg_LATEST.md
|
||||
# edit if needed, then:
|
||||
git commit -F .claude/git_commit_msg_LATEST.md
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
**Simple one-liner output:**
|
||||
```
|
||||
Add `MktPair.fqme` property for symbol resolution
|
||||
```
|
||||
|
||||
**Multi-file change output:**
|
||||
```
|
||||
Factor `.claude/skills/` into proper subdirs
|
||||
|
||||
Deats,
|
||||
- `commit_msg/` -> `commit-msg/` w/ enhanced
|
||||
frontmatter
|
||||
- all background skills set `user-invocable: false`
|
||||
- content split into supporting files
|
||||
|
||||
(this patch was generated in some part by
|
||||
[`claude-code`][claude-code-gh])
|
||||
[claude-code-gh]: https://github.com/anthropics/claude-code
|
||||
```
|
||||
|
||||
### Frontmatter Reference
|
||||
|
||||
The skill's `SKILL.md` uses these Claude Code
|
||||
frontmatter fields:
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: commit-msg
|
||||
description: >
|
||||
Generate piker-style git commit messages...
|
||||
argument-hint: "[optional-scope-or-description]"
|
||||
disable-model-invocation: true
|
||||
allowed-tools:
|
||||
- Bash(git *)
|
||||
- Read
|
||||
- Grep
|
||||
- Glob
|
||||
- Write
|
||||
---
|
||||
```
|
||||
|
||||
| Field | Purpose |
|
||||
|-------|---------|
|
||||
| `argument-hint` | Shows hint in autocomplete |
|
||||
| `disable-model-invocation` | Only user can trigger via `/commit-msg` |
|
||||
| `allowed-tools` | Tools the skill can use |
|
||||
|
||||
### Dynamic Context
|
||||
|
||||
The skill injects live data at invocation time
|
||||
via `!`backtick`` syntax in the `SKILL.md`:
|
||||
|
||||
```markdown
|
||||
## Current staged changes
|
||||
!`git diff --staged --stat`
|
||||
|
||||
## Recent commit style reference
|
||||
!`git log --oneline -10`
|
||||
```
|
||||
|
||||
This means the staged diff stats and recent log
|
||||
are always fresh when the skill runs -- no stale
|
||||
context.
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
################
|
||||
# ---- CEXY ----
|
||||
|
||||
################
|
||||
[binance]
|
||||
accounts.paper = 'paper'
|
||||
|
||||
|
|
@ -12,41 +13,28 @@ accounts.spot = 'spot'
|
|||
spot.use_testnet = false
|
||||
spot.api_key = ''
|
||||
spot.api_secret = ''
|
||||
# ------ binance ------
|
||||
|
||||
|
||||
[deribit]
|
||||
# std assets
|
||||
key_id = ''
|
||||
key_secret = ''
|
||||
# options
|
||||
accounts.option = 'option'
|
||||
option.use_testnet = false
|
||||
option.key_id = ''
|
||||
option.key_secret = ''
|
||||
# aux logging from `cryptofeed`
|
||||
option.log.filename = 'cryptofeed.log'
|
||||
option.log.level = 'DEBUG'
|
||||
option.log.disabled = true
|
||||
# ------ deribit ------
|
||||
|
||||
|
||||
[kraken]
|
||||
key_descr = ''
|
||||
api_key = ''
|
||||
secret = ''
|
||||
# ------ kraken ------
|
||||
|
||||
|
||||
[kucoin]
|
||||
key_id = ''
|
||||
key_secret = ''
|
||||
key_passphrase = ''
|
||||
# ------ kucoin ------
|
||||
|
||||
|
||||
################
|
||||
# -- BROKERZ ---
|
||||
|
||||
################
|
||||
[questrade]
|
||||
refresh_token = ''
|
||||
access_token = ''
|
||||
|
|
@ -54,55 +42,44 @@ api_server = 'https://api06.iq.questrade.com/'
|
|||
expires_in = 1800
|
||||
token_type = 'Bearer'
|
||||
expires_at = 1616095326.355846
|
||||
# ------ questrade ------
|
||||
|
||||
|
||||
[ib]
|
||||
# define the (set of) host-port socketaddrs that
|
||||
# brokerd.ib will scan to connect to an API endpoint
|
||||
# (ib-gw or ib-tws listening instances)
|
||||
hosts = [
|
||||
'127.0.0.1',
|
||||
]
|
||||
# XXX: the order in which ports will be scanned
|
||||
# (by the `brokerd` daemon-actor)
|
||||
# is determined # by the line order here.
|
||||
# TODO: when we eventually spawn gateways in our
|
||||
# container, we can just dynamically allocate these
|
||||
# using IBC.
|
||||
ports = [
|
||||
4002, # gw
|
||||
7497, # tws
|
||||
]
|
||||
|
||||
# When API endpoints are being scanned durin startup, the order
|
||||
# of user-defined-account "names" (as defined below) here
|
||||
# determines which py-client connection is given priority to be
|
||||
# used for data-feed-requests by according to whichever client
|
||||
# connected to an API endpoing which reported the equivalent
|
||||
# account number for that name.
|
||||
# XXX: for a paper account the flex web query service
|
||||
# is not supported so you have to manually download
|
||||
# and XML report and put it in a location that can be
|
||||
# accessed by the ``brokerd.ib`` backend code for parsing.
|
||||
flex_token = ''
|
||||
flex_trades_query_id = '' # live account
|
||||
|
||||
# when clients are being scanned this determines
|
||||
# which clients are preferred to be used for data
|
||||
# feeds based on the order of account names, if
|
||||
# detected as active on an API client.
|
||||
prefer_data_account = [
|
||||
'paper',
|
||||
'margin',
|
||||
'ira',
|
||||
]
|
||||
|
||||
# For long-term trades txn (transaction) history
|
||||
# processing (i.e your txn ledger with IB) you can
|
||||
# (automatically for live accounts) query the FLEX
|
||||
# report system for past history.
|
||||
#
|
||||
# (For paper accounts the web query service
|
||||
# is not supported so you have to manually download
|
||||
# an XML report and put it in a location that can be
|
||||
# accessed by our `brokerd.ib` backend code for parsing).
|
||||
#
|
||||
flex_token = ''
|
||||
flex_trades_query_id = '' # live account
|
||||
|
||||
# define "aliases" (names) for each account number
|
||||
# such that the names can be reffed and logged throughout
|
||||
# `piker.accounting` subsys and more easily
|
||||
# referred to by the user.
|
||||
#
|
||||
# These keys will be the set exposed through the order-mode
|
||||
# account-selection UI so that numbers are never shown.
|
||||
[ib.accounts]
|
||||
paper = 'DU0000000' # <- literal account #
|
||||
margin = 'U0000000'
|
||||
ira = 'U0000000'
|
||||
# ------ ib ------
|
||||
# the order in which accounts will be selectable
|
||||
# in the order mode UI (if found via clients during
|
||||
# API-app scanning)when a new symbol is loaded.
|
||||
paper = 'XX0000000'
|
||||
margin = 'X0000000'
|
||||
ira = 'X0000000'
|
||||
|
|
|
|||
|
|
@ -1,9 +1,7 @@
|
|||
[network]
|
||||
pikerd = [
|
||||
'/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
|
||||
# '/uds/6116', # TODO std uds socket file
|
||||
]
|
||||
|
||||
tsdb.backend = 'marketstore'
|
||||
tsdb.host = 'localhost'
|
||||
tsdb.grpc_port = 5995
|
||||
|
||||
[ui]
|
||||
# set custom font + size which will scale entire UI
|
||||
|
|
|
|||
135
default.nix
135
default.nix
|
|
@ -1,135 +0,0 @@
|
|||
with (import <nixpkgs> {});
|
||||
let
|
||||
glibStorePath = lib.getLib glib;
|
||||
zlibStorePath = lib.getLib zlib;
|
||||
zstdStorePath = lib.getLib zstd;
|
||||
dbusStorePath = lib.getLib dbus;
|
||||
libGLStorePath = lib.getLib libGL;
|
||||
freetypeStorePath = lib.getLib freetype;
|
||||
qt6baseStorePath = lib.getLib qt6.qtbase;
|
||||
fontconfigStorePath = lib.getLib fontconfig;
|
||||
libxkbcommonStorePath = lib.getLib libxkbcommon;
|
||||
xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
|
||||
|
||||
pypkgs = python313Packages;
|
||||
qtpyStorePath = lib.getLib pypkgs.qtpy;
|
||||
pyqt6StorePath = lib.getLib pypkgs.pyqt6;
|
||||
pyqt6SipStorePath = lib.getLib pypkgs.pyqt6-sip;
|
||||
rapidfuzzStorePath = lib.getLib pypkgs.rapidfuzz;
|
||||
qdarkstyleStorePath = lib.getLib pypkgs.qdarkstyle;
|
||||
|
||||
xorgLibX11StorePath = lib.getLib xorg.libX11;
|
||||
xorgLibxcbStorePath = lib.getLib xorg.libxcb;
|
||||
xorgxcbutilwmStorePath = lib.getLib xorg.xcbutilwm;
|
||||
xorgxcbutilimageStorePath = lib.getLib xorg.xcbutilimage;
|
||||
xorgxcbutilerrorsStorePath = lib.getLib xorg.xcbutilerrors;
|
||||
xorgxcbutilkeysymsStorePath = lib.getLib xorg.xcbutilkeysyms;
|
||||
xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
|
||||
in
|
||||
stdenv.mkDerivation {
|
||||
name = "piker-qt6-uv";
|
||||
buildInputs = [
|
||||
# System requirements.
|
||||
glib
|
||||
zlib
|
||||
dbus
|
||||
zstd
|
||||
libGL
|
||||
freetype
|
||||
qt6.qtbase
|
||||
libgcc.lib
|
||||
fontconfig
|
||||
libxkbcommon
|
||||
|
||||
# Xorg requirements
|
||||
xcb-util-cursor
|
||||
xorg.libxcb
|
||||
xorg.libX11
|
||||
xorg.xcbutilwm
|
||||
xorg.xcbutilimage
|
||||
xorg.xcbutilerrors
|
||||
xorg.xcbutilkeysyms
|
||||
xorg.xcbutilrenderutil
|
||||
|
||||
# Python requirements.
|
||||
python313
|
||||
uv
|
||||
pypkgs.qdarkstyle
|
||||
pypkgs.rapidfuzz
|
||||
pypkgs.pyqt6
|
||||
pypkgs.qtpy
|
||||
];
|
||||
src = null;
|
||||
shellHook = ''
|
||||
set -e
|
||||
|
||||
# Set the Qt plugin path
|
||||
# export QT_DEBUG_PLUGINS=1
|
||||
|
||||
QTBASE_PATH="${qt6baseStorePath}/lib"
|
||||
QT_PLUGIN_PATH="$QTBASE_PATH/qt-6/plugins"
|
||||
QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
|
||||
|
||||
LIB_GCC_PATH="${libgcc.lib}/lib"
|
||||
GLIB_PATH="${glibStorePath}/lib"
|
||||
ZSTD_PATH="${zstdStorePath}/lib"
|
||||
ZLIB_PATH="${zlibStorePath}/lib"
|
||||
DBUS_PATH="${dbusStorePath}/lib"
|
||||
LIBGL_PATH="${libGLStorePath}/lib"
|
||||
FREETYPE_PATH="${freetypeStorePath}/lib"
|
||||
FONTCONFIG_PATH="${fontconfigStorePath}/lib"
|
||||
LIB_XKB_COMMON_PATH="${libxkbcommonStorePath}/lib"
|
||||
|
||||
XCB_UTIL_CURSOR_PATH="${xcbutilcursorStorePath}/lib"
|
||||
XORG_LIB_X11_PATH="${xorgLibX11StorePath}/lib"
|
||||
XORG_LIB_XCB_PATH="${xorgLibxcbStorePath}/lib"
|
||||
XORG_XCB_UTIL_IMAGE_PATH="${xorgxcbutilimageStorePath}/lib"
|
||||
XORG_XCB_UTIL_WM_PATH="${xorgxcbutilwmStorePath}/lib"
|
||||
XORG_XCB_UTIL_RENDER_UTIL_PATH="${xorgxcbutilrenderutilStorePath}/lib"
|
||||
XORG_XCB_UTIL_KEYSYMS_PATH="${xorgxcbutilkeysymsStorePath}/lib"
|
||||
XORG_XCB_UTIL_ERRORS_PATH="${xorgxcbutilerrorsStorePath}/lib"
|
||||
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
|
||||
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_GCC_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$DBUS_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GLIB_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZLIB_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZSTD_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBGL_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FONTCONFIG_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FREETYPE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_XKB_COMMON_PATH"
|
||||
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XCB_UTIL_CURSOR_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_X11_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_XCB_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_IMAGE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_WM_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_RENDER_UTIL_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_KEYSYMS_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_ERRORS_PATH"
|
||||
|
||||
export LD_LIBRARY_PATH
|
||||
|
||||
RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.13/site-packages"
|
||||
QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.13/site-packages"
|
||||
QTPY_PATH="${qtpyStorePath}/lib/python3.13/site-packages"
|
||||
PYQT6_PATH="${pyqt6StorePath}/lib/python3.13/site-packages"
|
||||
PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.13/site-packages"
|
||||
|
||||
PATCH="$PATCH:$RPDFUZZ_PATH"
|
||||
PATCH="$PATCH:$QDRKSTYLE_PATH"
|
||||
PATCH="$PATCH:$QTPY_PATH"
|
||||
PATCH="$PATCH:$PYQT6_PATH"
|
||||
PATCH="$PATCH:$PYQT6_SIP_PATH"
|
||||
|
||||
export PATCH
|
||||
|
||||
# install all dev and extras
|
||||
uv sync --dev --all-extras
|
||||
|
||||
'';
|
||||
}
|
||||
37
develop.nix
37
develop.nix
|
|
@ -1,34 +1,28 @@
|
|||
with (import <nixpkgs> {});
|
||||
|
||||
with python310Packages;
|
||||
stdenv.mkDerivation {
|
||||
name = "poetry-env";
|
||||
name = "pip-env";
|
||||
buildInputs = [
|
||||
# System requirements.
|
||||
readline
|
||||
|
||||
# TODO: hacky non-poetry install stuff we need to get rid of!!
|
||||
poetry
|
||||
# virtualenv
|
||||
# setuptools
|
||||
# pip
|
||||
|
||||
# Python requirements (enough to get a virtualenv going).
|
||||
python311Full
|
||||
virtualenv
|
||||
setuptools
|
||||
pip
|
||||
|
||||
# obviously, and see below for hacked linking
|
||||
python311Packages.pyqt5
|
||||
python311Packages.pyqt5_sip
|
||||
# python311Packages.qtpy
|
||||
pyqt5
|
||||
|
||||
# Python requirements (enough to get a virtualenv going).
|
||||
python310Full
|
||||
|
||||
# numerics deps
|
||||
python311Packages.levenshtein
|
||||
python311Packages.fastparquet
|
||||
python311Packages.polars
|
||||
python310Packages.python-Levenshtein
|
||||
python310Packages.fastparquet
|
||||
python310Packages.polars
|
||||
|
||||
];
|
||||
# environment.sessionVariables = {
|
||||
# LD_LIBRARY_PATH = "${pkgs.stdenv.cc.cc.lib}/lib";
|
||||
# };
|
||||
src = null;
|
||||
shellHook = ''
|
||||
# Allow the use of wheels.
|
||||
|
|
@ -36,12 +30,13 @@ stdenv.mkDerivation {
|
|||
|
||||
# Augment the dynamic linker path
|
||||
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${R}/lib/R/lib:${readline}/lib
|
||||
|
||||
export QT_QPA_PLATFORM_PLUGIN_PATH="${qt5.qtbase.bin}/lib/qt-${qt5.qtbase.version}/plugins";
|
||||
|
||||
if [ ! -d ".venv" ]; then
|
||||
poetry install --with uis
|
||||
if [ ! -d "venv" ]; then
|
||||
virtualenv venv
|
||||
fi
|
||||
|
||||
poetry shell
|
||||
source venv/bin/activate
|
||||
'';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,138 +1,30 @@
|
|||
running ``ib`` gateway in ``docker``
|
||||
------------------------------------
|
||||
We have a config based on a well maintained community
|
||||
image from `@gnzsnz`:
|
||||
We have a config based on the (now defunct)
|
||||
image from "waytrade":
|
||||
|
||||
https://github.com/gnzsnz/ib-gateway-docker
|
||||
https://github.com/waytrade/ib-gateway-docker
|
||||
|
||||
|
||||
To startup this image simply run the command::
|
||||
To startup this image with our custom settings
|
||||
simply run the command::
|
||||
|
||||
docker compose up
|
||||
|
||||
(For further usage^ see the official `docker-compose`_ docs)
|
||||
And you should have the following socket-available services:
|
||||
|
||||
- ``x11vnc1@127.0.0.1:3003``
|
||||
- ``ib-gw@127.0.0.1:4002``
|
||||
|
||||
And you should have the following socket-available services by
|
||||
default:
|
||||
You can attach to the container via a VNC client
|
||||
without password auth.
|
||||
|
||||
- ``x11vnc1 @ 127.0.0.1:5900``
|
||||
- ``ib-gw @ 127.0.0.1:4002``
|
||||
|
||||
You can now attach to the container via a VNC client with password-auth;
|
||||
here is an example using ``vncclient`` on ``linux``::
|
||||
|
||||
vncviewer localhost:5900
|
||||
|
||||
now enter the pw (password) you set via an (see second code blob)
|
||||
`.env file`_ or pw-file according to the `credentials section`_.
|
||||
|
||||
If you want to change away from their default config see the example
|
||||
`docker-compose.yml`-config issue and config-section of the readme,
|
||||
|
||||
- https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
|
||||
- https://github.com/gnzsnz/ib-gateway-docker/discussions/103
|
||||
|
||||
.. _.env file: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#how-to-use-it
|
||||
.. _docker-compose: https://docs.docker.com/compose/
|
||||
.. _credentials section: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#credentials
|
||||
|
||||
|
||||
Connecting to the API from `piker`
|
||||
---------------------------------
|
||||
In order to expose the container's API endpoint to the
|
||||
`brokerd/datad/ib` actor, we need to add a section to the user's
|
||||
`brokers.toml` config (note the below is similar to the repo-shipped
|
||||
template file),
|
||||
|
||||
.. code:: toml
|
||||
|
||||
[ib]
|
||||
# define the (set of) host-port socketaddrs that
|
||||
# brokerd.ib will scan to connect to an API endpoint
|
||||
# (ib-gw or ib-tws listening instances)
|
||||
hosts = [
|
||||
'127.0.0.1',
|
||||
]
|
||||
ports = [
|
||||
4002, # gw
|
||||
7497, # tws
|
||||
]
|
||||
|
||||
# When API endpoints are being scanned durin startup, the order
|
||||
# of user-defined-account "names" (as defined below) here
|
||||
# determines which py-client connection is given priority to be
|
||||
# used for data-feed-requests by according to whichever client
|
||||
# connected to an API endpoing which reported the equivalent
|
||||
# account number for that name.
|
||||
prefer_data_account = [
|
||||
'paper',
|
||||
'margin',
|
||||
'ira',
|
||||
]
|
||||
|
||||
# define "aliases" (names) for each account number
|
||||
# such that the names can be reffed and logged throughout
|
||||
# `piker.accounting` subsys and more easily
|
||||
# referred to by the user.
|
||||
#
|
||||
# These keys will be the set exposed through the order-mode
|
||||
# account-selection UI so that numbers are never shown.
|
||||
[ib.accounts]
|
||||
paper = 'XX0000000'
|
||||
margin = 'X0000000'
|
||||
ira = 'X0000000'
|
||||
|
||||
|
||||
the broker daemon can also connect to the container's VNC server for
|
||||
added functionalies including,
|
||||
|
||||
- viewing the API endpoint program's GUI for manual interventions,
|
||||
- workarounds for historical data throttling using hotkey hacks,
|
||||
|
||||
Add a further section to `brokers.toml` which maps each API-ep's
|
||||
port to a table of VNC server connection info like,
|
||||
|
||||
.. code:: toml
|
||||
|
||||
[ib.vnc_addrs]
|
||||
4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
|
||||
|
||||
The `pw = 'doggy'` here ^ should the same value as the particular
|
||||
container instances `.env` file setting (when it was run),
|
||||
|
||||
.. code:: ini
|
||||
|
||||
VNC_SERVER_PASSWORD='doggy'
|
||||
|
||||
|
||||
IF you also want to run ``TWS``
|
||||
-------------------------------
|
||||
You can also run it containerized,
|
||||
|
||||
https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#using-tws
|
||||
|
||||
|
||||
SECURITY stuff (advanced, only if you're paranoid)
|
||||
--------------------------------------------------
|
||||
First and foremost if doing a "distributed" container setup where you
|
||||
run the ``ib-gw`` docker container and your connecting API client
|
||||
(likely ``ib_async`` from python) on **different hosts** be sure to
|
||||
read the `security considerations`_ section!
|
||||
|
||||
And for a further (somewhat paranoid) perspective from
|
||||
a long-time-ago serious devops eng..
|
||||
|
||||
Though "``ib``" claims they filter remote host connections outside
|
||||
``localhost`` (aka ``127.0.0.1`` on ipv4) it's prolly justified if
|
||||
you'd like to filter the socket at the *OS level* using a stateless
|
||||
firewall rule::
|
||||
SECURITY STUFF!?!?!
|
||||
-------------------
|
||||
Though "``ib``" claims they host filter connections outside
|
||||
localhost (aka ``127.0.0.1``) it's probably better if you filter
|
||||
the socket at the OS level using a stateless firewall rule::
|
||||
|
||||
ip rule add not unicast iif lo to 0.0.0.0/0 dport 4002
|
||||
|
||||
|
||||
We will soon have this either baked into our own custom derivative
|
||||
image (or patched into the current upstream one after further testin)
|
||||
but for now you'll have to do it urself, diggity dawg.
|
||||
|
||||
.. _security considerations: https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#security-considerations
|
||||
We will soon have this baked into our own custom image but for
|
||||
now you'll have to do it urself dawgy.
|
||||
|
|
|
|||
|
|
@ -1,15 +1,10 @@
|
|||
# a community maintained IB API container!
|
||||
#
|
||||
# https://github.com/gnzsnz/ib-gateway-docker
|
||||
#
|
||||
# For piker we (currently) include some minor deviations
|
||||
# for some config files in the `volumes` section.
|
||||
#
|
||||
# See full configuration settings @
|
||||
# - https://github.com/gnzsnz/ib-gateway-docker?tab=readme-ov-file#configuration
|
||||
# - https://github.com/gnzsnz/ib-gateway-docker/discussions/103
|
||||
# rework from the original @
|
||||
# https://github.com/waytrade/ib-gateway-docker/blob/master/docker-compose.yml
|
||||
version: "3.5"
|
||||
|
||||
|
||||
services:
|
||||
|
||||
ib_gw_paper:
|
||||
|
||||
# apparently java is a mega cukc:
|
||||
|
|
@ -24,9 +19,8 @@ services:
|
|||
|
||||
# other image tags available:
|
||||
# https://github.com/waytrade/ib-gateway-docker#supported-tags
|
||||
# image: waytrade/ib-gateway:1012.2i
|
||||
image: ghcr.io/gnzsnz/ib-gateway:latest
|
||||
|
||||
# image: waytrade/ib-gateway:981.3j
|
||||
image: waytrade/ib-gateway:1012.2i
|
||||
restart: 'no' # restart on boot whenev there's a crash or user clicsk
|
||||
network_mode: 'host'
|
||||
|
||||
|
|
@ -55,22 +49,16 @@ services:
|
|||
target: /root/scripts/run_x11_vnc.sh
|
||||
read_only: true
|
||||
|
||||
# NOTE: an alt method to fill these out is to
|
||||
# define an `.env` file in the same dir as
|
||||
# this compose file.
|
||||
# NOTE:to fill these out, define an `.env` file in the same dir as
|
||||
# this compose file which looks something like:
|
||||
# TWS_USERID='myuser'
|
||||
# TWS_PASSWORD='guest'
|
||||
environment:
|
||||
TWS_USERID: ${TWS_USERID}
|
||||
# TWS_USERID: 'myuser'
|
||||
TWS_PASSWORD: ${TWS_PASSWORD}
|
||||
# TWS_PASSWORD: 'guest'
|
||||
TRADING_MODE: ${TRADING_MODE}
|
||||
# TRADING_MODE: 'paper'
|
||||
VNC_SERVER_PASSWORD: ${VNC_SERVER_PASSWORD}
|
||||
# VNC_SERVER_PASSWORD: 'doggy'
|
||||
|
||||
# TODO, see if we can get this supported like it
|
||||
# was on the old `waytrade` image?
|
||||
# VNC_SERVER_PORT: '3003'
|
||||
TRADING_MODE: 'paper'
|
||||
VNC_SERVER_PASSWORD: 'doggy'
|
||||
VNC_SERVER_PORT: '3003'
|
||||
|
||||
# ports:
|
||||
# - target: 4002
|
||||
|
|
@ -87,9 +75,6 @@ services:
|
|||
# - "127.0.0.1:4002:4002"
|
||||
# - "127.0.0.1:5900:5900"
|
||||
|
||||
# TODO, a masked but working example of dual paper + live
|
||||
# ib-gw instances running in a single app run!
|
||||
#
|
||||
# ib_gw_live:
|
||||
# image: waytrade/ib-gateway:1012.2i
|
||||
# restart: no
|
||||
|
|
|
|||
|
|
@ -117,57 +117,9 @@ SecondFactorDevice=
|
|||
|
||||
# If you use the IBKR Mobile app for second factor authentication,
|
||||
# and you fail to complete the process before the time limit imposed
|
||||
# by IBKR, this setting tells IBC whether to automatically restart
|
||||
# the login sequence, giving you another opportunity to complete
|
||||
# second factor authentication.
|
||||
#
|
||||
# Permitted values are 'yes' and 'no'.
|
||||
#
|
||||
# If this setting is not present or has no value, then the value
|
||||
# of the deprecated ExitAfterSecondFactorAuthenticationTimeout is
|
||||
# used instead. If this also has no value, then this setting defaults
|
||||
# to 'no'.
|
||||
#
|
||||
# NB: you must be using IBC v3.14.0 or later to use this setting:
|
||||
# earlier versions ignore it.
|
||||
|
||||
ReloginAfterSecondFactorAuthenticationTimeout=
|
||||
|
||||
|
||||
# This setting is only relevant if
|
||||
# ReloginAfterSecondFactorAuthenticationTimeout is set to 'yes',
|
||||
# or if ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
|
||||
#
|
||||
# It controls how long (in seconds) IBC waits for login to complete
|
||||
# after the user acknowledges the second factor authentication
|
||||
# alert at the IBKR Mobile app. If login has not completed after
|
||||
# this time, IBC terminates.
|
||||
# The default value is 60.
|
||||
|
||||
SecondFactorAuthenticationExitInterval=
|
||||
|
||||
|
||||
# This setting specifies the timeout for second factor authentication
|
||||
# imposed by IB. The value is in seconds. You should not change this
|
||||
# setting unless you have reason to believe that IB has changed the
|
||||
# timeout. The default value is 180.
|
||||
|
||||
SecondFactorAuthenticationTimeout=180
|
||||
|
||||
|
||||
# DEPRECATED SETTING
|
||||
# ------------------
|
||||
#
|
||||
# ExitAfterSecondFactorAuthenticationTimeout - THIS SETTING WILL BE
|
||||
# REMOVED IN A FUTURE RELEASE. For IBC version 3.14.0 and later, see
|
||||
# the notes for ReloginAfterSecondFactorAuthenticationTimeout above.
|
||||
#
|
||||
# For IBC versions earlier than 3.14.0: If you use the IBKR Mobile
|
||||
# app for second factor authentication, and you fail to complete the
|
||||
# process before the time limit imposed by IBKR, you can use this
|
||||
# setting to tell IBC to exit: arrangements can then be made to
|
||||
# automatically restart IBC in order to initiate the login sequence
|
||||
# afresh. Otherwise, manual intervention at TWS's
|
||||
# by IBKR, you can use this setting to tell IBC to exit: arrangements
|
||||
# can then be made to automatically restart IBC in order to initiate
|
||||
# the login sequence afresh. Otherwise, manual intervention at TWS's
|
||||
# Second Factor Authentication dialog is needed to complete the
|
||||
# login.
|
||||
#
|
||||
|
|
@ -180,18 +132,29 @@ SecondFactorAuthenticationTimeout=180
|
|||
ExitAfterSecondFactorAuthenticationTimeout=no
|
||||
|
||||
|
||||
# This setting is only relevant if
|
||||
# ExitAfterSecondFactorAuthenticationTimeout is set to 'yes'.
|
||||
#
|
||||
# It controls how long (in seconds) IBC waits for login to complete
|
||||
# after the user acknowledges the second factor authentication
|
||||
# alert at the IBKR Mobile app. If login has not completed after
|
||||
# this time, IBC terminates.
|
||||
# The default value is 40.
|
||||
|
||||
SecondFactorAuthenticationExitInterval=
|
||||
|
||||
|
||||
# Trading Mode
|
||||
# ------------
|
||||
#
|
||||
# This indicates whether the live account or the paper trading
|
||||
# account corresponding to the supplied credentials is to be used.
|
||||
# The allowed values are 'live' (the default) and 'paper'.
|
||||
#
|
||||
# If this is set to 'live', then the credentials for the live
|
||||
# account must be supplied. If it is set to 'paper', then either
|
||||
# the live or the paper-trading credentials may be supplied.
|
||||
# TWS 955 introduced a new Trading Mode combo box on its login
|
||||
# dialog. This indicates whether the live account or the paper
|
||||
# trading account corresponding to the supplied credentials is
|
||||
# to be used. The allowed values are 'live' (the default) and
|
||||
# 'paper'. For earlier versions of TWS this setting has no
|
||||
# effect.
|
||||
|
||||
TradingMode=paper
|
||||
TradingMode=
|
||||
|
||||
|
||||
# Paper-trading Account Warning
|
||||
|
|
@ -225,7 +188,7 @@ AcceptNonBrokerageAccountWarning=yes
|
|||
#
|
||||
# The default value is 60.
|
||||
|
||||
LoginDialogDisplayTimeout=60
|
||||
LoginDialogDisplayTimeout=20
|
||||
|
||||
|
||||
|
||||
|
|
@ -254,15 +217,7 @@ LoginDialogDisplayTimeout=60
|
|||
# but they are acceptable.
|
||||
#
|
||||
# The default is the current working directory when IBC is
|
||||
# started, unless the TWS_SETTINGS_PATH setting in the relevant
|
||||
# start script is set.
|
||||
#
|
||||
# If both this setting and TWS_SETTINGS_PATH are set, then this
|
||||
# setting takes priority. Note that if they have different values,
|
||||
# auto-restart will not work.
|
||||
#
|
||||
# NB: this setting is now DEPRECATED. You should use the
|
||||
# TWS_SETTINGS_PATH setting in the relevant start script.
|
||||
# started.
|
||||
|
||||
IbDir=/root/Jts
|
||||
|
||||
|
|
@ -329,32 +284,15 @@ ExistingSessionDetectedAction=primary
|
|||
# Override TWS API Port Number
|
||||
# ----------------------------
|
||||
#
|
||||
# If OverrideTwsApiPort is set to an integer, IBC changes the
|
||||
# 'Socket port' in TWS's API configuration to that number shortly
|
||||
# after startup (but note that for the FIX Gateway, this setting is
|
||||
# actually stored in jts.ini rather than the Gateway's settings
|
||||
# file). Leaving the setting blank will make no change to
|
||||
# the current setting. This setting is only intended for use in
|
||||
# certain specialized situations where the port number needs to
|
||||
# be set dynamically at run-time, and for the FIX Gateway: most
|
||||
# non-FIX users will never need it, so don't use it unless you know
|
||||
# you need it.
|
||||
|
||||
OverrideTwsApiPort=4000
|
||||
|
||||
|
||||
# Override TWS Master Client ID
|
||||
# -----------------------------
|
||||
#
|
||||
# If OverrideTwsMasterClientID is set to an integer, IBC changes the
|
||||
# 'Master Client ID' value in TWS's API configuration to that
|
||||
# value shortly after startup. Leaving the setting blank will make
|
||||
# no change to the current setting. This setting is only intended
|
||||
# for use in certain specialized situations where the value needs to
|
||||
# If OverrideTwsApiPort is set to an integer, IBC changes the
|
||||
# 'Socket port' in TWS's API configuration to that number shortly
|
||||
# after startup. Leaving the setting blank will make no change to
|
||||
# the current setting. This setting is only intended for use in
|
||||
# certain specialized situations where the port number needs to
|
||||
# be set dynamically at run-time: most users will never need it,
|
||||
# so don't use it unless you know you need it.
|
||||
|
||||
OverrideTwsMasterClientID=
|
||||
; OverrideTwsApiPort=4002
|
||||
|
||||
|
||||
# Read-only Login
|
||||
|
|
@ -364,13 +302,11 @@ OverrideTwsMasterClientID=
|
|||
# account security programme, the user will not be asked to perform
|
||||
# the second factor authentication action, and login to TWS will
|
||||
# occur automatically in read-only mode: in this mode, placing or
|
||||
# managing orders is not allowed.
|
||||
#
|
||||
# If set to 'no', and the user is enrolled in IB's account security
|
||||
# programme, the second factor authentication process is handled
|
||||
# according to the Second Factor Authentication Settings described
|
||||
# elsewhere in this file.
|
||||
#
|
||||
# managing orders is not allowed. If set to 'no', and the user is
|
||||
# enrolled in IB's account security programme, the user must perform
|
||||
# the relevant second factor authentication action to complete the
|
||||
# login.
|
||||
|
||||
# If the user is not enrolled in IB's account security programme,
|
||||
# this setting is ignored. The default is 'no'.
|
||||
|
||||
|
|
@ -390,44 +326,7 @@ ReadOnlyLogin=no
|
|||
# set the relevant checkbox (this only needs to be done once) and
|
||||
# not provide a value for this setting.
|
||||
|
||||
ReadOnlyApi=
|
||||
|
||||
|
||||
# API Precautions
|
||||
# ---------------
|
||||
#
|
||||
# These settings relate to the corresponding 'Precautions' checkboxes in the
|
||||
# API section of the Global Configuration dialog.
|
||||
#
|
||||
# For all of these, the accepted values are:
|
||||
# - 'yes' sets the checkbox
|
||||
# - 'no' clears the checkbox
|
||||
# - if not set, the existing TWS/Gateway configuration is unchanged
|
||||
#
|
||||
# NB: thess settings are really only supplied for the benefit of new TWS
|
||||
# or Gateway instances that are being automatically installed and
|
||||
# started without user intervention, or where user settings are not preserved
|
||||
# between sessions (eg some Docker containers). Where a user is involved, they
|
||||
# should use the Global Configuration to set the relevant checkboxes and not
|
||||
# provide values for these settings.
|
||||
|
||||
BypassOrderPrecautions=
|
||||
|
||||
BypassBondWarning=
|
||||
|
||||
BypassNegativeYieldToWorstConfirmation=
|
||||
|
||||
BypassCalledBondWarning=
|
||||
|
||||
BypassSameActionPairTradeWarning=
|
||||
|
||||
BypassPriceBasedVolatilityRiskWarning=
|
||||
|
||||
BypassUSStocksMarketDataInSharesWarning=
|
||||
|
||||
BypassRedirectOrderWarning=
|
||||
|
||||
BypassNoOverfillProtectionPrecaution=
|
||||
ReadOnlyApi=no
|
||||
|
||||
|
||||
# Market data size for US stocks - lots or shares
|
||||
|
|
@ -482,145 +381,54 @@ AcceptBidAskLastSizeDisplayUpdateNotification=accept
|
|||
SendMarketDataInLotsForUSstocks=
|
||||
|
||||
|
||||
# Trusted API Client IPs
|
||||
# ----------------------
|
||||
#
|
||||
# NB: THIS SETTING IS ONLY RELEVANT FOR THE GATEWAY, AND ONLY WHEN FIX=yes.
|
||||
# In all other cases it is ignored.
|
||||
#
|
||||
# This is a list of IP addresses separated by commas. API clients with IP
|
||||
# addresses in this list are able to connect to the API without Gateway
|
||||
# generating the 'Incoming connection' popup.
|
||||
#
|
||||
# Note that 127.0.0.1 is always permitted to connect, so do not include it
|
||||
# in this setting.
|
||||
|
||||
TrustedTwsApiClientIPs=
|
||||
|
||||
|
||||
# Reset Order ID Sequence
|
||||
# -----------------------
|
||||
#
|
||||
# The setting resets the order id sequence for orders submitted via the API, so
|
||||
# that the next invocation of the `NextValidId` API callback will return the
|
||||
# value 1. The reset occurs when TWS starts.
|
||||
#
|
||||
# Note that order ids are reset for all API clients, except those that have
|
||||
# outstanding (ie incomplete) orders: their order id sequence carries on as
|
||||
# before.
|
||||
#
|
||||
# Valid values are 'yes', 'true', 'false' and 'no'. The default is 'no'.
|
||||
|
||||
ResetOrderIdsAtStart=
|
||||
|
||||
|
||||
# This setting specifies IBC's action when TWS displays the dialog asking for
|
||||
# confirmation of a request to reset the API order id sequence.
|
||||
#
|
||||
# Note that the Gateway never displays this dialog, so this setting is ignored
|
||||
# for a Gateway session.
|
||||
#
|
||||
# Valid values consist of two strings separated by a solidus '/'. The first
|
||||
# value specifies the action to take when the order id reset request resulted
|
||||
# from setting ResetOrderIdsAtStart=yes. The second specifies the action to
|
||||
# take when the order id reset request is a result of the user clicking the
|
||||
# 'Reset API order ID sequence' button in the API configuration. Each value
|
||||
# must be one of the following:
|
||||
#
|
||||
# 'confirm'
|
||||
# order ids will be reset
|
||||
#
|
||||
# 'reject'
|
||||
# order ids will not be reset
|
||||
#
|
||||
# 'ignore'
|
||||
# IBC will ignore the dialog. The user must take action.
|
||||
#
|
||||
# The default setting is ignore/ignore
|
||||
|
||||
# Examples:
|
||||
#
|
||||
# 'confirm/reject' - confirm order id reset only if ResetOrderIdsAtStart=yes
|
||||
# and reject any user-initiated requests
|
||||
#
|
||||
# 'ignore/confirm' - user must decide what to do if ResetOrderIdsAtStart=yes
|
||||
# and confirm user-initiated requests
|
||||
#
|
||||
# 'reject/ignore' - reject order id reset if ResetOrderIdsAtStart=yes but
|
||||
# allow user to handle user-initiated requests
|
||||
|
||||
ConfirmOrderIdReset=
|
||||
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 4. TWS Auto-Logoff and Auto-Restart
|
||||
# 4. TWS Auto-Closedown
|
||||
# =============================================================================
|
||||
#
|
||||
# TWS and Gateway insist on being restarted every day. Two alternative
|
||||
# automatic options are offered:
|
||||
# IMPORTANT NOTE: Starting with TWS 974, this setting no longer
|
||||
# works properly, because IB have changed the way TWS handles its
|
||||
# autologoff mechanism.
|
||||
#
|
||||
# - Auto-Logoff: at a specified time, TWS shuts down tidily, without
|
||||
# restarting.
|
||||
# You should now configure the TWS autologoff time to something
|
||||
# convenient for you, and restart IBC each day.
|
||||
#
|
||||
# - Auto-Restart: at a specified time, TWS shuts down and then restarts
|
||||
# without the user having to re-autheticate.
|
||||
#
|
||||
# The normal way to configure the time at which this happens is via the Lock
|
||||
# and Exit section of the Configuration dialog. Once this time has been
|
||||
# configured in this way, the setting persists until the user changes it again.
|
||||
#
|
||||
# However, there are situations where there is no user available to do this
|
||||
# configuration, or where there is no persistent storage (for example some
|
||||
# Docker images). In such cases, the auto-restart or auto-logoff time can be
|
||||
# set whenever IBC starts with the settings below.
|
||||
#
|
||||
# The value, if specified, must be a time in HH:MM AM/PM format, for example
|
||||
# 08:00 AM or 10:00 PM. Note that there must be a single space between the
|
||||
# two parts of this value; also that midnight is "12:00 AM" and midday is
|
||||
# "12:00 PM".
|
||||
#
|
||||
# If no value is specified for either setting, the currently configured
|
||||
# settings will apply. If a value is supplied for one setting, the other
|
||||
# setting is cleared. If values are supplied for both settings, only the
|
||||
# auto-restart time is set, and the auto-logoff time is cleared.
|
||||
#
|
||||
# Note that for a normal TWS/Gateway installation with persistent storage
|
||||
# (for example on a desktop computer) the value will be persisted as if the
|
||||
# user had set it via the configuration dialog.
|
||||
#
|
||||
# If you choose to auto-restart, you should take note of the considerations
|
||||
# described at the link below. Note that where this information mentions
|
||||
# 'manual authentication', restarting IBC will do the job (IBKR does not
|
||||
# recognise the existence of IBC in its docuemntation).
|
||||
#
|
||||
# https://www.interactivebrokers.com/en/software/tws/twsguide.htm#usersguidebook/configuretws/auto_restart_info.htm
|
||||
#
|
||||
# If you use the "RESTART" command via the IBC command server, and IBC is
|
||||
# running any version of the Gateway (or a version of TWS earlier than 1018),
|
||||
# note that this will set the Auto-Restart time in Gateway/TWS's configuration
|
||||
# dialog to the time at which the restart actually happens (which may be up to
|
||||
# a minute after the RESTART command is issued). To prevent future auto-
|
||||
# restarts at this time, you must make sure you have set AutoLogoffTime or
|
||||
# AutoRestartTime to your desired value before running IBC. NB: this does not
|
||||
# apply to TWS from version 1018 onwards.
|
||||
# Alternatively, discontinue use of IBC and use the auto-relogin
|
||||
# mechanism within TWS 974 and later versions (note that the
|
||||
# auto-relogin mechanism provided by IB is not available if you
|
||||
# use IBC).
|
||||
|
||||
AutoLogoffTime=
|
||||
# Set to yes or no (lower case).
|
||||
#
|
||||
# yes means allow TWS to shut down automatically at its
|
||||
# specified shutdown time, which is set via the TWS
|
||||
# configuration menu.
|
||||
#
|
||||
# no means TWS never shuts down automatically.
|
||||
#
|
||||
# NB: IB recommends that you do not keep TWS running
|
||||
# continuously. If you set this setting to 'no', you may
|
||||
# experience incorrect TWS operation.
|
||||
#
|
||||
# NB: the default for this setting is 'no'. Since this will
|
||||
# only work properly with TWS versions earlier than 974, you
|
||||
# should explicitly set this to 'yes' for version 974 and later.
|
||||
|
||||
IbAutoClosedown=yes
|
||||
|
||||
AutoRestartTime=
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 5. TWS Tidy Closedown Time
|
||||
# =============================================================================
|
||||
#
|
||||
# Specifies a time at which TWS will close down tidily, with no restart.
|
||||
# NB: starting with TWS 974 this is no longer a useful option
|
||||
# because both TWS and Gateway now have the same auto-logoff
|
||||
# mechanism, and IBC can no longer avoid this.
|
||||
#
|
||||
# There is little reason to use this setting. It is similar to AutoLogoffTime,
|
||||
# but can include a day-of-the-week, whereas AutoLogoffTime and AutoRestartTime
|
||||
# apply every day. So for example you could use ClosedownAt in conjunction with
|
||||
# AutoRestartTime to shut down TWS on Friday evenings after the markets
|
||||
# close, without it running on Saturday as well.
|
||||
# Note that giving this setting a value does not change TWS's
|
||||
# auto-logoff in any way: any setting will be additional to the
|
||||
# TWS auto-logoff.
|
||||
#
|
||||
# To tell IBC to tidily close TWS at a specified time every
|
||||
# day, set this value to <hh:mm>, for example:
|
||||
|
|
@ -679,7 +487,7 @@ AcceptIncomingConnectionAction=reject
|
|||
# no means the dialog remains on display and must be
|
||||
# handled by the user.
|
||||
|
||||
AllowBlindTrading=no
|
||||
AllowBlindTrading=yes
|
||||
|
||||
|
||||
# Save Settings on a Schedule
|
||||
|
|
@ -722,26 +530,6 @@ AllowBlindTrading=no
|
|||
SaveTwsSettingsAt=
|
||||
|
||||
|
||||
# Confirm Crypto Currency Orders Automatically
|
||||
# --------------------------------------------
|
||||
#
|
||||
# When you place an order for a cryptocurrency contract, a dialog is displayed
|
||||
# asking you to confirm that you want to place the order, and notifying you
|
||||
# that you are placing an order to trade cryptocurrency with Paxos, a New York
|
||||
# limited trust company, and not at Interactive Brokers.
|
||||
#
|
||||
# transmit means that the order will be placed automatically, and the
|
||||
# dialog will then be closed
|
||||
#
|
||||
# cancel means that the order will not be placed, and the dialog will
|
||||
# then be closed
|
||||
#
|
||||
# manual means that IBC will take no action and the user must deal
|
||||
# with the dialog
|
||||
|
||||
ConfirmCryptoCurrencyOrders=transmit
|
||||
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 7. Settings Specific to Indian Versions of TWS
|
||||
|
|
@ -778,17 +566,13 @@ DismissNSEComplianceNotice=yes
|
|||
#
|
||||
# The port number that IBC listens on for commands
|
||||
# such as "STOP". DO NOT set this to the port number
|
||||
# used for TWS API connections.
|
||||
#
|
||||
# The convention is to use 7462 for this port,
|
||||
# but it must be set to a different value from any other
|
||||
# IBC instance that might run at the same time.
|
||||
#
|
||||
# The default value is 0, which tells IBC not to start
|
||||
# the command server
|
||||
# used for TWS API connections. There is no good reason
|
||||
# to change this setting unless the port is used by
|
||||
# some other application (typically another instance of
|
||||
# IBC). The default value is 0, which tells IBC not to
|
||||
# start the command server
|
||||
|
||||
#CommandServerPort=7462
|
||||
CommandServerPort=0
|
||||
|
||||
|
||||
# Permitted Command Sources
|
||||
|
|
@ -799,19 +583,19 @@ CommandServerPort=0
|
|||
# IBC. Commands can always be sent from the
|
||||
# same host as IBC is running on.
|
||||
|
||||
ControlFrom=
|
||||
ControlFrom=127.0.0.1
|
||||
|
||||
|
||||
# Address for Receiving Commands
|
||||
# ------------------------------
|
||||
#
|
||||
# Specifies the IP address on which the Command Server
|
||||
# is to listen. For a multi-homed host, this can be used
|
||||
# is so listen. For a multi-homed host, this can be used
|
||||
# to specify that connection requests are only to be
|
||||
# accepted on the specified address. The default is to
|
||||
# accept connection requests on all local addresses.
|
||||
|
||||
BindAddress=
|
||||
BindAddress=127.0.0.1
|
||||
|
||||
|
||||
# Command Prompt
|
||||
|
|
@ -837,7 +621,7 @@ CommandPrompt=
|
|||
# information is sent. The default is that such information
|
||||
# is not sent.
|
||||
|
||||
SuppressInfoMessages=yes
|
||||
SuppressInfoMessages=no
|
||||
|
||||
|
||||
|
||||
|
|
@ -867,10 +651,10 @@ SuppressInfoMessages=yes
|
|||
# The LogStructureScope setting indicates which windows are
|
||||
# eligible for structure logging:
|
||||
#
|
||||
# - (default value) if set to 'known', only windows that
|
||||
# IBC recognizes are eligible - these are windows that
|
||||
# IBC has some interest in monitoring, usually to take
|
||||
# some action on the user's behalf;
|
||||
# - if set to 'known', only windows that IBC recognizes
|
||||
# are eligible - these are windows that IBC has some
|
||||
# interest in monitoring, usually to take some action
|
||||
# on the user's behalf;
|
||||
#
|
||||
# - if set to 'unknown', only windows that IBC does not
|
||||
# recognize are eligible. Most windows displayed by
|
||||
|
|
@ -883,8 +667,9 @@ SuppressInfoMessages=yes
|
|||
# - if set to 'all', then every window displayed by TWS
|
||||
# is eligible.
|
||||
#
|
||||
# The default value is 'known'.
|
||||
|
||||
LogStructureScope=known
|
||||
LogStructureScope=all
|
||||
|
||||
|
||||
# When to Log Window Structure
|
||||
|
|
@ -897,15 +682,13 @@ LogStructureScope=known
|
|||
# structure of an eligible window the first time it
|
||||
# is encountered;
|
||||
#
|
||||
# - if set to 'openclose', the structure is logged every
|
||||
# time an eligible window is opened or closed;
|
||||
#
|
||||
# - if set to 'activate', the structure is logged every
|
||||
# time an eligible window is made active;
|
||||
#
|
||||
# - (default value) if set to 'never' or 'no' or 'false',
|
||||
# structure information is never logged.
|
||||
# - if set to 'never' or 'no' or 'false', structure
|
||||
# information is never logged.
|
||||
#
|
||||
# The default value is 'never'.
|
||||
|
||||
LogStructureWhen=never
|
||||
|
||||
|
|
@ -925,3 +708,4 @@ LogStructureWhen=never
|
|||
#LogComponents=
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -121,7 +121,6 @@ async def bot_main():
|
|||
# tick_throttle=10,
|
||||
) as feed,
|
||||
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
assert accounts
|
||||
|
|
|
|||
27
flake.lock
27
flake.lock
|
|
@ -1,27 +0,0 @@
|
|||
{
|
||||
"nodes": {
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1765779637,
|
||||
"narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
|
||||
"owner": "nixos",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nixos",
|
||||
"ref": "nixos-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"root": {
|
||||
"inputs": {
|
||||
"nixpkgs": "nixpkgs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"root": "root",
|
||||
"version": 7
|
||||
}
|
||||
103
flake.nix
103
flake.nix
|
|
@ -1,103 +0,0 @@
|
|||
# An "impure" template thx to `pyproject.nix`,
|
||||
# https://pyproject-nix.github.io/pyproject.nix/templates.html#impure
|
||||
# https://github.com/pyproject-nix/pyproject.nix/blob/master/templates/impure/flake.nix
|
||||
{
|
||||
description = "An impure `piker` overlay using `uv` with Nix(OS)";
|
||||
|
||||
inputs = {
|
||||
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
|
||||
};
|
||||
|
||||
outputs =
|
||||
{ nixpkgs, ... }:
|
||||
let
|
||||
inherit (nixpkgs) lib;
|
||||
forAllSystems = lib.genAttrs lib.systems.flakeExposed;
|
||||
in
|
||||
{
|
||||
devShells = forAllSystems (
|
||||
system:
|
||||
let
|
||||
pkgs = nixpkgs.legacyPackages.${system};
|
||||
|
||||
# do store-path extractions
|
||||
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
||||
# ?TODO? can remove below since manual linking not needed?
|
||||
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
||||
|
||||
# XXX NOTE XXX, for now we overlay specific pkgs via
|
||||
# a major-version-pinned-`cpython`
|
||||
cpython = "python313";
|
||||
pypkgs = pkgs."${cpython}Packages";
|
||||
in
|
||||
{
|
||||
default = pkgs.mkShell {
|
||||
|
||||
packages = with pkgs; [
|
||||
# XXX, ensure sh completions active!
|
||||
bashInteractive
|
||||
bash-completion
|
||||
|
||||
# dev utils
|
||||
ruff
|
||||
pypkgs.ruff
|
||||
|
||||
qt6.qtwayland
|
||||
qt6.qtbase
|
||||
|
||||
uv
|
||||
python313 # ?TODO^ how to set from `cpython` above?
|
||||
pypkgs.pyqt6
|
||||
pypkgs.pyqt6-sip
|
||||
pypkgs.qtpy
|
||||
pypkgs.qdarkstyle
|
||||
pypkgs.rapidfuzz
|
||||
];
|
||||
|
||||
shellHook = ''
|
||||
# unmask to debug **this** dev-shell-hook
|
||||
# set -e
|
||||
|
||||
# set qt-base/plugin path(s)
|
||||
QTBASE_PATH="${qt6baseStorePath}/lib"
|
||||
QT_PLUGIN_PATH="${qt6baseStorePath}/lib/qt-6/plugins"
|
||||
QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
|
||||
|
||||
# link in Qt cc lib paths from <nixpkgs>
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
|
||||
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
|
||||
|
||||
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
||||
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
||||
|
||||
export LD_LIBRARY_PATH
|
||||
|
||||
# RUNTIME-SETTINGS
|
||||
#
|
||||
# ------ Qt ------
|
||||
# XXX, unmask to debug qt .so linking/loading deats
|
||||
# export QT_DEBUG_PLUGINS=1
|
||||
#
|
||||
# ALSO, for *modern linux* DEs,
|
||||
# - maybe set wayland-mode (TODO, parametrtize this!)
|
||||
# * a chosen wayland-mode shell-integration
|
||||
export QT_QPA_PLATFORM="wayland"
|
||||
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
||||
|
||||
# ------ uv ------
|
||||
# - always use the ./py313/ venv-subdir
|
||||
export UV_PROJECT_ENVIRONMENT="py313"
|
||||
# sync project-env with all extras
|
||||
uv sync --dev --all-extras --no-group lint
|
||||
|
||||
# ------ TIPS ------
|
||||
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
||||
# run the `nix develop` cmd with,
|
||||
# >> nix develop -c uv run xonsh
|
||||
'';
|
||||
};
|
||||
}
|
||||
);
|
||||
};
|
||||
}
|
||||
|
|
@ -19,10 +19,8 @@
|
|||
for tendiez.
|
||||
|
||||
'''
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ..log import get_logger
|
||||
|
||||
from .calc import (
|
||||
iter_by_dt,
|
||||
)
|
||||
|
|
@ -35,6 +33,7 @@ from ._pos import (
|
|||
Account,
|
||||
load_account,
|
||||
load_account_from_ledger,
|
||||
open_pps,
|
||||
open_account,
|
||||
Position,
|
||||
)
|
||||
|
|
@ -43,6 +42,7 @@ from ._mktinfo import (
|
|||
dec_digits,
|
||||
digits_to_dec,
|
||||
MktPair,
|
||||
Symbol,
|
||||
unpack_fqme,
|
||||
_derivs as DerivTypes,
|
||||
)
|
||||
|
|
@ -53,23 +53,14 @@ from ._allocate import (
|
|||
|
||||
|
||||
log = get_logger(__name__)
|
||||
# ?TODO, enable console on import
|
||||
# [ ] necessary? or `open_brokerd_dialog()` doing it is sufficient?
|
||||
#
|
||||
# bc might as well enable whenev imported by
|
||||
# other sub-sys code (namely `.clearing`).
|
||||
get_console_log(
|
||||
level='warning',
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# TODO, the `as <samename>` style?
|
||||
__all__ = [
|
||||
'Account',
|
||||
'Allocator',
|
||||
'Asset',
|
||||
'MktPair',
|
||||
'Position',
|
||||
'Symbol',
|
||||
'Transaction',
|
||||
'TransactionLedger',
|
||||
'dec_digits',
|
||||
|
|
@ -79,6 +70,7 @@ __all__ = [
|
|||
'load_account_from_ledger',
|
||||
'mk_allocator',
|
||||
'open_account',
|
||||
'open_pps',
|
||||
'open_trade_ledger',
|
||||
'unpack_fqme',
|
||||
'DerivTypes',
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ import tomli_w # for fast ledger writing
|
|||
|
||||
from piker.types import Struct
|
||||
from piker import config
|
||||
from piker.log import get_logger
|
||||
from ..log import get_logger
|
||||
from .calc import (
|
||||
iter_by_dt,
|
||||
)
|
||||
|
|
@ -239,9 +239,7 @@ class TransactionLedger(UserDict):
|
|||
|
||||
symcache: SymbologyCache = self._symcache
|
||||
towrite: dict[str, Any] = {}
|
||||
for tid, txdict in self.tx_sort(
|
||||
self.data.copy()
|
||||
):
|
||||
for tid, txdict in self.tx_sort(self.data.copy()):
|
||||
# write blank-str expiry for non-expiring assets
|
||||
if (
|
||||
'expiry' in txdict
|
||||
|
|
@ -379,7 +377,7 @@ def open_trade_ledger(
|
|||
account,
|
||||
dirpath=_fp,
|
||||
)
|
||||
cpy: dict = ledger_dict.copy()
|
||||
cpy = ledger_dict.copy()
|
||||
|
||||
# XXX NOTE: if not provided presume we are being called from
|
||||
# sync code and need to maybe run `trio` to generate..
|
||||
|
|
@ -408,13 +406,7 @@ def open_trade_ledger(
|
|||
account=account,
|
||||
mod=mod,
|
||||
symcache=symcache,
|
||||
|
||||
# NOTE: allow backends to provide custom ledger sorting
|
||||
tx_sort=getattr(
|
||||
mod,
|
||||
'tx_sort',
|
||||
tx_sort,
|
||||
),
|
||||
tx_sort=getattr(mod, 'tx_sort', tx_sort),
|
||||
)
|
||||
try:
|
||||
yield ledger
|
||||
|
|
|
|||
|
|
@ -305,8 +305,8 @@ class MktPair(Struct, frozen=True):
|
|||
# config right?
|
||||
# src_type: AssetTypeName
|
||||
|
||||
# for derivs, info describing contract, egs. strike price, call
|
||||
# or put, swap type, exercise model, etc.
|
||||
# for derivs, info describing contract, egs.
|
||||
# strike price, call or put, swap type, exercise model, etc.
|
||||
contract_info: list[str] | None = None
|
||||
|
||||
# TODO: rename to sectype since all of these can
|
||||
|
|
@ -327,11 +327,7 @@ class MktPair(Struct, frozen=True):
|
|||
) -> dict:
|
||||
d = super().to_dict(**kwargs)
|
||||
d['src'] = self.src.to_dict(**kwargs)
|
||||
|
||||
if not isinstance(self.dst, str):
|
||||
d['dst'] = self.dst.to_dict(**kwargs)
|
||||
else:
|
||||
d['dst'] = str(self.dst)
|
||||
d['dst'] = self.dst.to_dict(**kwargs)
|
||||
|
||||
d['price_tick'] = str(self.price_tick)
|
||||
d['size_tick'] = str(self.size_tick)
|
||||
|
|
@ -353,16 +349,11 @@ class MktPair(Struct, frozen=True):
|
|||
Constructor for a received msg-dict normally received over IPC.
|
||||
|
||||
'''
|
||||
if not isinstance(
|
||||
dst_asset_msg := msg.pop('dst'),
|
||||
str,
|
||||
):
|
||||
dst: Asset = Asset.from_msg(dst_asset_msg) # .copy()
|
||||
else:
|
||||
dst: str = dst_asset_msg
|
||||
dst_asset_msg = msg.pop('dst')
|
||||
dst = Asset.from_msg(dst_asset_msg) # .copy()
|
||||
|
||||
src_asset_msg: dict = msg.pop('src')
|
||||
src: Asset = Asset.from_msg(src_asset_msg) # .copy()
|
||||
src_asset_msg = msg.pop('src')
|
||||
src = Asset.from_msg(src_asset_msg) # .copy()
|
||||
|
||||
# XXX NOTE: ``msgspec`` can encode `Decimal` but it doesn't
|
||||
# decide to it by default since we aren't spec-cing these
|
||||
|
|
@ -390,8 +381,8 @@ class MktPair(Struct, frozen=True):
|
|||
cls,
|
||||
fqme: str,
|
||||
|
||||
price_tick: float|str,
|
||||
size_tick: float|str,
|
||||
price_tick: float | str,
|
||||
size_tick: float | str,
|
||||
bs_mktid: str,
|
||||
|
||||
broker: str | None = None,
|
||||
|
|
@ -677,3 +668,90 @@ def unpack_fqme(
|
|||
# '.'.join([mkt_ep, venue]),
|
||||
suffix,
|
||||
)
|
||||
|
||||
|
||||
class Symbol(Struct):
|
||||
'''
|
||||
I guess this is some kinda container thing for dealing with
|
||||
all the different meta-data formats from brokers?
|
||||
|
||||
'''
|
||||
key: str
|
||||
|
||||
broker: str = ''
|
||||
venue: str = ''
|
||||
|
||||
# precision descriptors for price and vlm
|
||||
tick_size: Decimal = Decimal('0.01')
|
||||
lot_tick_size: Decimal = Decimal('0.0')
|
||||
|
||||
suffix: str = ''
|
||||
broker_info: dict[str, dict[str, Any]] = {}
|
||||
|
||||
@classmethod
|
||||
def from_fqme(
|
||||
cls,
|
||||
fqsn: str,
|
||||
info: dict[str, Any],
|
||||
|
||||
) -> Symbol:
|
||||
broker, mktep, venue, suffix = unpack_fqme(fqsn)
|
||||
tick_size = info.get('price_tick_size', 0.01)
|
||||
lot_size = info.get('lot_tick_size', 0.0)
|
||||
|
||||
return Symbol(
|
||||
broker=broker,
|
||||
key=mktep,
|
||||
tick_size=tick_size,
|
||||
lot_tick_size=lot_size,
|
||||
venue=venue,
|
||||
suffix=suffix,
|
||||
broker_info={broker: info},
|
||||
)
|
||||
|
||||
@property
|
||||
def type_key(self) -> str:
|
||||
return list(self.broker_info.values())[0]['asset_type']
|
||||
|
||||
@property
|
||||
def tick_size_digits(self) -> int:
|
||||
return float_digits(self.tick_size)
|
||||
|
||||
@property
|
||||
def lot_size_digits(self) -> int:
|
||||
return float_digits(self.lot_tick_size)
|
||||
|
||||
@property
|
||||
def price_tick(self) -> Decimal:
|
||||
return Decimal(str(self.tick_size))
|
||||
|
||||
@property
|
||||
def size_tick(self) -> Decimal:
|
||||
return Decimal(str(self.lot_tick_size))
|
||||
|
||||
@property
|
||||
def broker(self) -> str:
|
||||
return list(self.broker_info.keys())[0]
|
||||
|
||||
@property
|
||||
def fqme(self) -> str:
|
||||
return maybe_cons_tokens([
|
||||
self.key, # final "pair name" (eg. qqq[/usd], btcusdt)
|
||||
self.venue,
|
||||
self.suffix, # includes expiry and other con info
|
||||
self.broker,
|
||||
])
|
||||
|
||||
def quantize(
|
||||
self,
|
||||
size: float,
|
||||
) -> Decimal:
|
||||
digits = float_digits(self.lot_tick_size)
|
||||
return Decimal(size).quantize(
|
||||
Decimal(f'1.{"0".ljust(digits, "0")}'),
|
||||
rounding=ROUND_HALF_EVEN
|
||||
)
|
||||
|
||||
# NOTE: when cast to `str` return fqme
|
||||
def __str__(self) -> str:
|
||||
return self.fqme
|
||||
|
|
|
|||
|
|
@ -30,8 +30,7 @@ from types import ModuleType
|
|||
from typing import (
|
||||
Any,
|
||||
Iterator,
|
||||
Generator,
|
||||
TYPE_CHECKING,
|
||||
Generator
|
||||
)
|
||||
|
||||
import pendulum
|
||||
|
|
@ -60,16 +59,10 @@ from ..clearing._messages import (
|
|||
BrokerdPosition,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
from piker.data._symcache import SymbologyCache
|
||||
from ..log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.data._symcache import SymbologyCache
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
class Position(Struct):
|
||||
|
|
@ -360,20 +353,17 @@ class Position(Struct):
|
|||
) -> bool:
|
||||
'''
|
||||
Update clearing table by calculating the rolling ppu and
|
||||
(accumulative) size in both the clears entry and local attrs
|
||||
state.
|
||||
(accumulative) size in both the clears entry and local
|
||||
attrs state.
|
||||
|
||||
Inserts are always done in datetime sorted order.
|
||||
|
||||
'''
|
||||
# added: bool = False
|
||||
tid: str = t.tid
|
||||
if tid in self._events:
|
||||
log.debug(
|
||||
f'Txn is already added?\n'
|
||||
f'\n'
|
||||
f'{t}\n'
|
||||
)
|
||||
return False
|
||||
log.warning(f'{t} is already added?!')
|
||||
# return added
|
||||
|
||||
# TODO: apparently this IS possible with a dict but not
|
||||
# common and probably not that beneficial unless we're also
|
||||
|
|
@ -454,12 +444,6 @@ class Position(Struct):
|
|||
# def suggest_split(self) -> float:
|
||||
# ...
|
||||
|
||||
# ?TODO, for sending rendered state over the wire?
|
||||
# def summary(self) -> PositionSummary:
|
||||
# do minimal conversion to a subset of fields
|
||||
# currently defined in `.clearing._messages.BrokerdPosition`
|
||||
|
||||
|
||||
|
||||
class Account(Struct):
|
||||
'''
|
||||
|
|
@ -503,23 +487,12 @@ class Account(Struct):
|
|||
|
||||
def update_from_ledger(
|
||||
self,
|
||||
ledger: TransactionLedger|dict[str, Transaction],
|
||||
ledger: TransactionLedger | dict[str, Transaction],
|
||||
cost_scalar: float = 2,
|
||||
symcache: SymbologyCache|None = None,
|
||||
symcache: SymbologyCache | None = None,
|
||||
|
||||
_mktmap_table: dict[str, MktPair] | None = None,
|
||||
|
||||
only_require: list[str]|True = True,
|
||||
# ^list of fqmes that are "required" to be processed from
|
||||
# this ledger pass; we often don't care about others and
|
||||
# definitely shouldn't always error in such cases.
|
||||
# (eg. broker backend loaded that doesn't yet supsport the
|
||||
# symcache but also, inside the paper engine we don't ad-hoc
|
||||
# request `get_mkt_info()` for every symbol in the ledger,
|
||||
# only the one for which we're simulating against).
|
||||
# TODO, not sure if there's a better soln for this, ideally
|
||||
# all backends get symcache support afap i guess..
|
||||
|
||||
) -> dict[str, Position]:
|
||||
'''
|
||||
Update the internal `.pps[str, Position]` table from input
|
||||
|
|
@ -562,40 +535,14 @@ class Account(Struct):
|
|||
if _mktmap_table is None:
|
||||
raise
|
||||
|
||||
required: bool = (
|
||||
only_require is True
|
||||
or (
|
||||
only_require is not True
|
||||
and
|
||||
fqme in only_require
|
||||
)
|
||||
)
|
||||
# XXX: caller is allowed to provide a fallback
|
||||
# mktmap table for the case where a new position is
|
||||
# being added and the preloaded symcache didn't
|
||||
# have this entry prior (eg. with frickin IB..)
|
||||
if (
|
||||
not (mkt := _mktmap_table.get(fqme))
|
||||
and
|
||||
required
|
||||
):
|
||||
raise
|
||||
|
||||
elif not required:
|
||||
continue
|
||||
|
||||
else:
|
||||
# should be an entry retreived somewhere
|
||||
assert mkt
|
||||
|
||||
mkt = _mktmap_table[fqme]
|
||||
|
||||
if not (pos := pps.get(bs_mktid)):
|
||||
|
||||
assert isinstance(
|
||||
mkt,
|
||||
MktPair,
|
||||
)
|
||||
|
||||
# if no existing pos, allocate fresh one.
|
||||
pos = pps[bs_mktid] = Position(
|
||||
mkt=mkt,
|
||||
|
|
@ -704,7 +651,7 @@ class Account(Struct):
|
|||
def write_config(self) -> None:
|
||||
'''
|
||||
Write the current account state to the user's account TOML file, normally
|
||||
something like `pps.toml`.
|
||||
something like ``pps.toml``.
|
||||
|
||||
'''
|
||||
# TODO: show diff output?
|
||||
|
|
@ -744,7 +691,7 @@ class Account(Struct):
|
|||
else:
|
||||
# TODO: we reallly need a diff set of
|
||||
# loglevels/colors per subsys.
|
||||
log.debug(
|
||||
log.warning(
|
||||
f'Recent position for {fqme} was closed!'
|
||||
)
|
||||
|
||||
|
|
@ -758,7 +705,7 @@ class Account(Struct):
|
|||
# XXX WTF: if we use a tomlkit.Integer here we get this
|
||||
# super weird --1 thing going on for cumsize!?1!
|
||||
# NOTE: the fix was to always float() the size value loaded
|
||||
# in open_account() below!
|
||||
# in open_pps() below!
|
||||
config.write(
|
||||
config=self.conf,
|
||||
path=self.conf_path,
|
||||
|
|
@ -942,6 +889,7 @@ def open_account(
|
|||
clears_table['dt'] = dt
|
||||
trans.append(Transaction(
|
||||
fqme=bs_mktid,
|
||||
# sym=mkt,
|
||||
bs_mktid=bs_mktid,
|
||||
tid=tid,
|
||||
# XXX: not sure why sometimes these are loaded as
|
||||
|
|
@ -964,22 +912,11 @@ def open_account(
|
|||
):
|
||||
expiry: pendulum.DateTime = pendulum.parse(expiry)
|
||||
|
||||
# !XXX, should never be duplicates over
|
||||
# a backend-(broker)-system's unique market-IDs!
|
||||
if pos := pp_objs.get(bs_mktid):
|
||||
if mkt != pos.mkt:
|
||||
log.warning(
|
||||
f'Duplicated position but diff `MktPair.fqme` ??\n'
|
||||
f'bs_mktid: {bs_mktid!r}\n'
|
||||
f'pos.mkt: {pos.mkt}\n'
|
||||
f'mkt: {mkt}\n'
|
||||
)
|
||||
else:
|
||||
pos = pp_objs[bs_mktid] = Position(
|
||||
mkt,
|
||||
split_ratio=split_ratio,
|
||||
bs_mktid=bs_mktid,
|
||||
)
|
||||
pp = pp_objs[bs_mktid] = Position(
|
||||
mkt,
|
||||
split_ratio=split_ratio,
|
||||
bs_mktid=bs_mktid,
|
||||
)
|
||||
|
||||
# XXX: super critical, we need to be sure to include
|
||||
# all pps.toml clears to avoid reusing clears that were
|
||||
|
|
@ -987,13 +924,8 @@ def open_account(
|
|||
# state, since today's records may have already been
|
||||
# processed!
|
||||
for t in trans:
|
||||
added: bool = pos.add_clear(t)
|
||||
if not added:
|
||||
log.warning(
|
||||
f'Txn already recorded in pp ??\n'
|
||||
f'\n'
|
||||
f'{t}\n'
|
||||
)
|
||||
pp.add_clear(t)
|
||||
|
||||
try:
|
||||
yield acnt
|
||||
finally:
|
||||
|
|
@ -1001,6 +933,20 @@ def open_account(
|
|||
acnt.write_config()
|
||||
|
||||
|
||||
# TODO: drop the old name and THIS!
|
||||
@cm
|
||||
def open_pps(
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> Generator[Account, None, None]:
|
||||
log.warning(
|
||||
'`open_pps()` is now deprecated!\n'
|
||||
'Please use `with open_account() as cnt:`'
|
||||
)
|
||||
with open_account(*args, **kwargs) as acnt:
|
||||
yield acnt
|
||||
|
||||
|
||||
def load_account_from_ledger(
|
||||
|
||||
brokername: str,
|
||||
|
|
|
|||
|
|
@ -22,9 +22,7 @@ you know when you're losing money (if possible) XD
|
|||
from __future__ import annotations
|
||||
from collections.abc import ValuesView
|
||||
from contextlib import contextmanager as cm
|
||||
from functools import partial
|
||||
from math import copysign
|
||||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
Callable,
|
||||
|
|
@ -32,7 +30,6 @@ from typing import (
|
|||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
from tractor.devx import maybe_open_crash_handler
|
||||
import polars as pl
|
||||
from pendulum import (
|
||||
DateTime,
|
||||
|
|
@ -40,16 +37,12 @@ from pendulum import (
|
|||
parse,
|
||||
)
|
||||
|
||||
from ..log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ._ledger import (
|
||||
Transaction,
|
||||
TransactionLedger,
|
||||
)
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def ppu(
|
||||
clears: Iterator[Transaction],
|
||||
|
|
@ -245,9 +238,6 @@ def iter_by_dt(
|
|||
|
||||
def dyn_parse_to_dt(
|
||||
tx: tuple[str, dict[str, Any]] | Transaction,
|
||||
|
||||
debug: bool = False,
|
||||
_invalid: list|None = None,
|
||||
) -> DateTime:
|
||||
|
||||
# handle `.items()` inputs
|
||||
|
|
@ -260,90 +250,33 @@ def iter_by_dt(
|
|||
# get best parser for this record..
|
||||
for k in parsers:
|
||||
if (
|
||||
(v := getattr(tx, k, None))
|
||||
or
|
||||
(
|
||||
isdict
|
||||
and
|
||||
(v := tx.get(k))
|
||||
)
|
||||
isdict and k in tx
|
||||
or getattr(tx, k, None)
|
||||
):
|
||||
v = tx[k] if isdict else tx.dt
|
||||
assert v is not None, f'No valid value for `{k}`!?'
|
||||
|
||||
# only call parser on the value if not None from
|
||||
# the `parsers` table above (when NOT using
|
||||
# `.get()`), otherwise pass through the value and
|
||||
# sort on it directly
|
||||
if (
|
||||
not isinstance(v, DateTime)
|
||||
and
|
||||
(parser := parsers.get(k))
|
||||
and (parser := parsers.get(k))
|
||||
):
|
||||
ret = parser(v)
|
||||
return parser(v)
|
||||
else:
|
||||
ret = v
|
||||
return v
|
||||
|
||||
return ret
|
||||
|
||||
else:
|
||||
log.debug(
|
||||
f'Parser-field not found in txn\n'
|
||||
f'\n'
|
||||
f'parser-field: {k!r}\n'
|
||||
f'txn: {tx!r}\n'
|
||||
f'\n'
|
||||
f'Trying next..\n'
|
||||
)
|
||||
continue
|
||||
|
||||
# XXX: we should never really get here bc it means some kinda
|
||||
# bad txn-record (field) data..
|
||||
#
|
||||
# -> set the `debug_mode = True` if you want to trace such
|
||||
# cases from REPL ;)
|
||||
else:
|
||||
# XXX: we should really never get here..
|
||||
# only if a ledger record has no expected sort(able)
|
||||
# field will we likely hit this.. like with ze IB.
|
||||
# if no sortable field just deliver epoch?
|
||||
log.warning(
|
||||
'No (time) sortable field for TXN:\n'
|
||||
f'{tx!r}\n'
|
||||
)
|
||||
report: str = (
|
||||
f'No supported time-field found in txn !?\n'
|
||||
f'\n'
|
||||
f'supported-time-fields: {parsers!r}\n'
|
||||
f'\n'
|
||||
f'txn: {tx!r}\n'
|
||||
)
|
||||
if debug:
|
||||
with maybe_open_crash_handler(
|
||||
pdb=debug,
|
||||
raise_on_exit=False,
|
||||
):
|
||||
raise ValueError(report)
|
||||
else:
|
||||
log.error(report)
|
||||
# XXX: should never get here..
|
||||
breakpoint()
|
||||
|
||||
if _invalid is not None:
|
||||
_invalid.append(tx)
|
||||
return from_timestamp(0.)
|
||||
|
||||
entry: tuple[str, dict]|Transaction
|
||||
invalid: list = []
|
||||
entry: tuple[str, dict] | Transaction
|
||||
for entry in sorted(
|
||||
records,
|
||||
key=key or partial(
|
||||
dyn_parse_to_dt,
|
||||
_invalid=invalid,
|
||||
),
|
||||
key=key or dyn_parse_to_dt,
|
||||
):
|
||||
if entry in invalid:
|
||||
log.warning(
|
||||
f'Ignoring txn w invalid timestamp ??\n'
|
||||
f'{pformat(entry)}\n'
|
||||
)
|
||||
continue
|
||||
|
||||
# NOTE the type sig above; either pairs or txns B)
|
||||
yield entry
|
||||
|
||||
|
|
@ -406,7 +339,6 @@ def open_ledger_dfs(
|
|||
acctname: str,
|
||||
|
||||
ledger: TransactionLedger | None = None,
|
||||
debug_mode: bool = False,
|
||||
|
||||
**kwargs,
|
||||
|
||||
|
|
@ -421,10 +353,8 @@ def open_ledger_dfs(
|
|||
can update the ledger on exit.
|
||||
|
||||
'''
|
||||
with maybe_open_crash_handler(
|
||||
pdb=debug_mode,
|
||||
# raise_on_exit=False,
|
||||
):
|
||||
from tractor._debug import open_crash_handler
|
||||
with open_crash_handler():
|
||||
if not ledger:
|
||||
import time
|
||||
from ._ledger import open_trade_ledger
|
||||
|
|
@ -516,7 +446,7 @@ def ledger_to_dfs(
|
|||
|
||||
df = dfs[key] = ldf.with_columns([
|
||||
|
||||
pl.cum_sum('size').alias('cumsize'),
|
||||
pl.cumsum('size').alias('cumsize'),
|
||||
|
||||
# amount of source asset "sent" (via buy txns in
|
||||
# the market) to acquire the dst asset, PER txn.
|
||||
|
|
@ -531,7 +461,7 @@ def ledger_to_dfs(
|
|||
]).with_columns([
|
||||
|
||||
# rolling balance in src asset units
|
||||
(pl.col('dst_bot').cum_sum() * -1).alias('src_balance'),
|
||||
(pl.col('dst_bot').cumsum() * -1).alias('src_balance'),
|
||||
|
||||
# "position operation type" in terms of increasing the
|
||||
# amount in the dst asset (entering) or decreasing the
|
||||
|
|
@ -673,7 +603,7 @@ def ledger_to_dfs(
|
|||
# cost that was included in the least-recently
|
||||
# entered txn that is still part of the current CSi
|
||||
# set.
|
||||
# => we look up the cost-per-unit cum_sum and apply
|
||||
# => we look up the cost-per-unit cumsum and apply
|
||||
# if over the current txn size (by multiplication)
|
||||
# and then reverse that previusly applied cost on
|
||||
# the txn_cost for this record.
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ CLI front end for trades ledger and position tracking management.
|
|||
from __future__ import annotations
|
||||
from pprint import pformat
|
||||
|
||||
|
||||
from rich.console import Console
|
||||
from rich.markdown import Markdown
|
||||
import polars as pl
|
||||
|
|
@ -28,10 +29,7 @@ import tractor
|
|||
import trio
|
||||
import typer
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ..log import get_logger
|
||||
from ..service import (
|
||||
open_piker_runtime,
|
||||
)
|
||||
|
|
@ -47,7 +45,6 @@ from .calc import (
|
|||
open_ledger_dfs,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
ledger = typer.Typer()
|
||||
|
||||
|
|
@ -82,10 +79,7 @@ def sync(
|
|||
"-l",
|
||||
),
|
||||
):
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
log = get_logger(loglevel)
|
||||
console = Console()
|
||||
|
||||
pair: tuple[str, str]
|
||||
|
|
@ -306,8 +300,7 @@ def disect(
|
|||
assert not df.is_empty()
|
||||
|
||||
# muck around in pdbp REPL
|
||||
# tractor.devx.mk_pdb().set_trace()
|
||||
# breakpoint()
|
||||
breakpoint()
|
||||
|
||||
# TODO: we REALLY need a better console REPL for this
|
||||
# kinda thing..
|
||||
|
|
|
|||
|
|
@ -25,16 +25,15 @@ from types import ModuleType
|
|||
|
||||
from tractor.trionics import maybe_open_context
|
||||
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
from ._util import (
|
||||
log,
|
||||
BrokerError,
|
||||
SymbolNotFound,
|
||||
NoData,
|
||||
DataUnavailable,
|
||||
DataThrottle,
|
||||
resproc,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
__all__: list[str] = [
|
||||
|
|
@ -44,13 +43,14 @@ __all__: list[str] = [
|
|||
'DataUnavailable',
|
||||
'DataThrottle',
|
||||
'resproc',
|
||||
'get_logger',
|
||||
]
|
||||
|
||||
__brokers__: list[str] = [
|
||||
'binance',
|
||||
'ib',
|
||||
'kraken',
|
||||
'kucoin',
|
||||
'kucoin'
|
||||
|
||||
# broken but used to work
|
||||
# 'questrade',
|
||||
|
|
@ -65,17 +65,13 @@ __brokers__: list[str] = [
|
|||
# bitso
|
||||
]
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
|
||||
def get_brokermod(brokername: str) -> ModuleType:
|
||||
'''
|
||||
Return the imported broker module by name.
|
||||
|
||||
'''
|
||||
module: ModuleType = import_module('.' + brokername, 'piker.brokers')
|
||||
module = import_module('.' + brokername, 'piker.brokers')
|
||||
# we only allow monkeying because it's for internal keying
|
||||
module.name = module.__name__.split('.')[-1]
|
||||
return module
|
||||
|
|
@ -102,14 +98,13 @@ async def open_cached_client(
|
|||
If one has not been setup do it and cache it.
|
||||
|
||||
'''
|
||||
brokermod: ModuleType = get_brokermod(brokername)
|
||||
|
||||
# TODO: make abstract or `typing.Protocol`
|
||||
# client: Client
|
||||
brokermod = get_brokermod(brokername)
|
||||
async with maybe_open_context(
|
||||
acm_func=brokermod.get_client,
|
||||
kwargs=kwargs,
|
||||
|
||||
) as (cache_hit, client):
|
||||
|
||||
if cache_hit:
|
||||
log.runtime(f'Reusing existing {client}')
|
||||
|
||||
|
|
|
|||
|
|
@ -33,18 +33,12 @@ import exceptiongroup as eg
|
|||
import tractor
|
||||
import trio
|
||||
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from . import _util
|
||||
from . import get_brokermod
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..data import _FeedsBus
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
# `brokerd` enabled modules
|
||||
# TODO: move this def to the `.data` subpkg..
|
||||
# NOTE: keeping this list as small as possible is part of our caps-sec
|
||||
|
|
@ -65,7 +59,7 @@ _data_mods: str = [
|
|||
async def _setup_persistent_brokerd(
|
||||
ctx: tractor.Context,
|
||||
brokername: str,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -78,14 +72,13 @@ async def _setup_persistent_brokerd(
|
|||
# since all hosted daemon tasks will reference this same
|
||||
# log instance's (actor local) state and thus don't require
|
||||
# any further (level) configuration on their own B)
|
||||
actor: tractor.Actor = tractor.current_actor()
|
||||
tll: str = actor.loglevel
|
||||
log = get_console_log(
|
||||
level=loglevel or tll,
|
||||
log = _util.get_console_log(
|
||||
loglevel or tractor.current_actor().loglevel,
|
||||
name=f'{_util.subsys}.{brokername}',
|
||||
with_tractor_log=bool(tll),
|
||||
)
|
||||
assert log.name == _util.subsys
|
||||
|
||||
# set global for this actor to this new process-wide instance B)
|
||||
_util.log = log
|
||||
|
||||
# further, set the log level on any broker broker specific
|
||||
# logger instance.
|
||||
|
|
@ -103,10 +96,7 @@ async def _setup_persistent_brokerd(
|
|||
# - `open_symbol_search()`
|
||||
# NOTE: see ep invocation details inside `.data.feed`.
|
||||
try:
|
||||
async with (
|
||||
# tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as service_nursery
|
||||
):
|
||||
async with trio.open_nursery() as service_nursery:
|
||||
bus: _FeedsBus = feed.get_feed_bus(
|
||||
brokername,
|
||||
service_nursery,
|
||||
|
|
@ -189,6 +179,9 @@ def broker_init(
|
|||
subpath: str = f'{modpath}.{submodname}'
|
||||
enabled.append(subpath)
|
||||
|
||||
# TODO XXX: DO WE NEED THIS?
|
||||
# enabled.append('piker.data.feed')
|
||||
|
||||
return (
|
||||
brokermod,
|
||||
start_actor_kwargs, # to `ActorNursery.start_actor()`
|
||||
|
|
@ -200,6 +193,7 @@ def broker_init(
|
|||
|
||||
|
||||
async def spawn_brokerd(
|
||||
|
||||
brokername: str,
|
||||
loglevel: str | None = None,
|
||||
|
||||
|
|
@ -207,10 +201,8 @@ async def spawn_brokerd(
|
|||
|
||||
) -> bool:
|
||||
|
||||
log.info(
|
||||
f'Spawning broker-daemon,\n'
|
||||
f'backend: {brokername!r}'
|
||||
)
|
||||
from piker.service._util import log # use service mngr log
|
||||
log.info(f'Spawning {brokername} broker daemon')
|
||||
|
||||
(
|
||||
brokermode,
|
||||
|
|
@ -257,7 +249,7 @@ async def spawn_brokerd(
|
|||
async def maybe_spawn_brokerd(
|
||||
|
||||
brokername: str,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
**pikerd_kwargs,
|
||||
|
||||
|
|
@ -273,7 +265,8 @@ async def maybe_spawn_brokerd(
|
|||
from piker.service import maybe_spawn_daemon
|
||||
|
||||
async with maybe_spawn_daemon(
|
||||
service_name=f'brokerd.{brokername}',
|
||||
|
||||
f'brokerd.{brokername}',
|
||||
service_task_target=spawn_brokerd,
|
||||
spawn_args={
|
||||
'brokername': brokername,
|
||||
|
|
|
|||
|
|
@ -18,14 +18,15 @@
|
|||
Handy cross-broker utils.
|
||||
|
||||
"""
|
||||
from __future__ import annotations
|
||||
# from functools import partial
|
||||
from functools import partial
|
||||
|
||||
import json
|
||||
import httpx
|
||||
import asks
|
||||
import logging
|
||||
|
||||
from piker.log import (
|
||||
from ..log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
colorize_json,
|
||||
)
|
||||
subsys: str = 'piker.brokers'
|
||||
|
|
@ -33,22 +34,12 @@ subsys: str = 'piker.brokers'
|
|||
# NOTE: level should be reset by any actor that is spawned
|
||||
# as well as given a (more) explicit name/key such
|
||||
# as `piker.brokers.binance` matching the subpkg.
|
||||
# log = get_logger(subsys)
|
||||
log = get_logger(subsys)
|
||||
|
||||
# ?TODO?? we could use this approach, but we need to be able
|
||||
# to pass multiple `name=` values so for example we can include the
|
||||
# emissions in `.accounting._pos` and others!
|
||||
# [ ] maybe we could do the `log = get_logger()` above,
|
||||
# then cycle through the list of subsys mods we depend on
|
||||
# and then get all their loggers and pass them to
|
||||
# `get_console_log(logger=)`??
|
||||
# [ ] OR just write THIS `get_console_log()` as a hook which does
|
||||
# that based on who calls it?.. i dunno
|
||||
#
|
||||
# get_console_log = partial(
|
||||
# get_console_log,
|
||||
# name=subsys,
|
||||
# )
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
name=subsys,
|
||||
)
|
||||
|
||||
|
||||
class BrokerError(Exception):
|
||||
|
|
@ -59,7 +50,6 @@ class SymbolNotFound(BrokerError):
|
|||
"Symbol not found by broker search"
|
||||
|
||||
|
||||
# TODO: these should probably be moved to `.tsp/.data`?
|
||||
class NoData(BrokerError):
|
||||
'''
|
||||
Symbol data not permitted or no data
|
||||
|
|
@ -69,15 +59,14 @@ class NoData(BrokerError):
|
|||
def __init__(
|
||||
self,
|
||||
*args,
|
||||
info: dict|None = None,
|
||||
frame_size: int = 1000,
|
||||
|
||||
) -> None:
|
||||
super().__init__(*args)
|
||||
self.info: dict|None = info
|
||||
|
||||
# when raised, machinery can check if the backend
|
||||
# set a "frame size" for doing datetime calcs.
|
||||
# self.frame_size: int = 1000
|
||||
self.frame_size: int = 1000
|
||||
|
||||
|
||||
class DataUnavailable(BrokerError):
|
||||
|
|
@ -99,18 +88,16 @@ class DataThrottle(BrokerError):
|
|||
|
||||
|
||||
def resproc(
|
||||
resp: httpx.Response,
|
||||
resp: asks.response_objects.Response,
|
||||
log: logging.Logger,
|
||||
return_json: bool = True,
|
||||
log_resp: bool = False,
|
||||
|
||||
) -> httpx.Response:
|
||||
'''
|
||||
Process response and return its json content.
|
||||
) -> asks.response_objects.Response:
|
||||
"""Process response and return its json content.
|
||||
|
||||
Raise the appropriate error on non-200 OK responses.
|
||||
|
||||
'''
|
||||
"""
|
||||
if not resp.status_code == 200:
|
||||
raise BrokerError(resp.body)
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C)
|
||||
# Guillermo Rodriguez (aka ze jefe)
|
||||
# Tyler Goodlet
|
||||
# (in stewardship for pikers)
|
||||
# Guillermo Rodriguez (aka ze jefe)
|
||||
# Tyler Goodlet
|
||||
# (in stewardship for pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
|
|
@ -25,7 +25,6 @@ from __future__ import annotations
|
|||
from collections import ChainMap
|
||||
from contextlib import (
|
||||
asynccontextmanager as acm,
|
||||
AsyncExitStack,
|
||||
)
|
||||
from datetime import datetime
|
||||
from pprint import pformat
|
||||
|
|
@ -42,7 +41,8 @@ import trio
|
|||
from pendulum import (
|
||||
now,
|
||||
)
|
||||
import httpx
|
||||
import asks
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
import numpy as np
|
||||
|
||||
from piker import config
|
||||
|
|
@ -52,13 +52,9 @@ from piker.clearing._messages import (
|
|||
from piker.accounting import (
|
||||
Asset,
|
||||
digits_to_dec,
|
||||
MktPair,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.data import (
|
||||
def_iohlcv_fields,
|
||||
match_from_pairs,
|
||||
)
|
||||
from piker.data import def_iohlcv_fields
|
||||
from piker.brokers import (
|
||||
resproc,
|
||||
SymbolNotFound,
|
||||
|
|
@ -68,6 +64,7 @@ from .venues import (
|
|||
PAIRTYPES,
|
||||
Pair,
|
||||
MarketType,
|
||||
|
||||
_spot_url,
|
||||
_futes_url,
|
||||
_testnet_futes_url,
|
||||
|
|
@ -77,18 +74,16 @@ from .venues import (
|
|||
log = get_logger('piker.brokers.binance')
|
||||
|
||||
|
||||
def get_config() -> dict[str, Any]:
|
||||
def get_config() -> dict:
|
||||
|
||||
conf: dict
|
||||
path: Path
|
||||
conf, path = config.load(
|
||||
conf_name='brokers',
|
||||
touch_if_dne=True,
|
||||
)
|
||||
section: dict = conf.get('binance')
|
||||
conf, path = config.load(touch_if_dne=True)
|
||||
|
||||
section = conf.get('binance')
|
||||
|
||||
if not section:
|
||||
log.warning(
|
||||
f'No config section found for binance in {path}'
|
||||
)
|
||||
log.warning(f'No config section found for binance in {path}')
|
||||
return {}
|
||||
|
||||
return section
|
||||
|
|
@ -144,7 +139,7 @@ def binance_timestamp(
|
|||
|
||||
class Client:
|
||||
'''
|
||||
Async ReST API client using `trio` + `httpx` B)
|
||||
Async ReST API client using ``trio`` + ``asks`` B)
|
||||
|
||||
Supports all of the spot, margin and futures endpoints depending
|
||||
on method.
|
||||
|
|
@ -153,17 +148,10 @@ class Client:
|
|||
def __init__(
|
||||
self,
|
||||
|
||||
venue_sessions: dict[
|
||||
str, # venue key
|
||||
tuple[httpx.AsyncClient, str] # session, eps path
|
||||
],
|
||||
conf: dict[str, Any],
|
||||
# TODO: change this to `Client.[mkt_]venue: MarketType`?
|
||||
mkt_mode: MarketType = 'spot',
|
||||
|
||||
) -> None:
|
||||
self.conf = conf
|
||||
|
||||
# build out pair info tables for each market type
|
||||
# and wrap in a chain-map view for search / query.
|
||||
self._spot_pairs: dict[str, Pair] = {} # spot info table
|
||||
|
|
@ -190,13 +178,44 @@ class Client:
|
|||
# market symbols for use by search. See `.exch_info()`.
|
||||
self._pairs: ChainMap[str, Pair] = ChainMap()
|
||||
|
||||
# spot EPs sesh
|
||||
self._sesh = asks.Session(connections=4)
|
||||
self._sesh.base_location: str = _spot_url
|
||||
# spot testnet
|
||||
self._test_sesh: asks.Session = asks.Session(connections=4)
|
||||
self._test_sesh.base_location: str = _testnet_spot_url
|
||||
|
||||
# margin and extended spot endpoints session.
|
||||
self._sapi_sesh = asks.Session(connections=4)
|
||||
self._sapi_sesh.base_location: str = _spot_url
|
||||
|
||||
# futes EPs sesh
|
||||
self._fapi_sesh = asks.Session(connections=4)
|
||||
self._fapi_sesh.base_location: str = _futes_url
|
||||
# futes testnet
|
||||
self._test_fapi_sesh: asks.Session = asks.Session(connections=4)
|
||||
self._test_fapi_sesh.base_location: str = _testnet_futes_url
|
||||
|
||||
# global client "venue selection" mode.
|
||||
# set this when you want to switch venues and not have to
|
||||
# specify the venue for the next request.
|
||||
self.mkt_mode: MarketType = mkt_mode
|
||||
|
||||
# per-mkt-venue API client table
|
||||
self.venue_sesh = venue_sessions
|
||||
# per 8
|
||||
self.venue_sesh: dict[
|
||||
str, # venue key
|
||||
tuple[asks.Session, str] # session, eps path
|
||||
] = {
|
||||
'spot': (self._sesh, '/api/v3/'),
|
||||
'spot_testnet': (self._test_sesh, '/fapi/v1/'),
|
||||
|
||||
'margin': (self._sapi_sesh, '/sapi/v1/'),
|
||||
|
||||
'usdtm_futes': (self._fapi_sesh, '/fapi/v1/'),
|
||||
'usdtm_futes_testnet': (self._test_fapi_sesh, '/fapi/v1/'),
|
||||
|
||||
# 'futes_coin': self._dapi, # TODO
|
||||
}
|
||||
|
||||
# lookup for going from `.mkt_mode: str` to the config
|
||||
# subsection `key: str`
|
||||
|
|
@ -211,6 +230,40 @@ class Client:
|
|||
'futes': ['usdtm_futes'],
|
||||
}
|
||||
|
||||
# for creating API keys see,
|
||||
# https://www.binance.com/en/support/faq/how-to-create-api-keys-on-binance-360002502072
|
||||
self.conf: dict = get_config()
|
||||
|
||||
for key, subconf in self.conf.items():
|
||||
if api_key := subconf.get('api_key', ''):
|
||||
venue_keys: list[str] = self.confkey2venuekeys[key]
|
||||
|
||||
venue_key: str
|
||||
sesh: asks.Session
|
||||
for venue_key in venue_keys:
|
||||
sesh, _ = self.venue_sesh[venue_key]
|
||||
|
||||
api_key_header: dict = {
|
||||
# taken from official:
|
||||
# https://github.com/binance/binance-futures-connector-python/blob/main/binance/api.py#L47
|
||||
"Content-Type": "application/json;charset=utf-8",
|
||||
|
||||
# TODO: prolly should just always query and copy
|
||||
# in the real latest ver?
|
||||
"User-Agent": "binance-connector/6.1.6smbz6",
|
||||
"X-MBX-APIKEY": api_key,
|
||||
}
|
||||
sesh.headers.update(api_key_header)
|
||||
|
||||
# if `.use_tesnet = true` in the config then
|
||||
# also add headers for the testnet session which
|
||||
# will be used for all order control
|
||||
if subconf.get('use_testnet', False):
|
||||
testnet_sesh, _ = self.venue_sesh[
|
||||
venue_key + '_testnet'
|
||||
]
|
||||
testnet_sesh.headers.update(api_key_header)
|
||||
|
||||
def _mk_sig(
|
||||
self,
|
||||
data: dict,
|
||||
|
|
@ -229,6 +282,7 @@ class Client:
|
|||
'to define the creds for auth-ed endpoints!?'
|
||||
)
|
||||
|
||||
|
||||
# XXX: Info on security and authentification
|
||||
# https://binance-docs.github.io/apidocs/#endpoint-security-type
|
||||
if not (api_secret := subconf.get('api_secret')):
|
||||
|
|
@ -257,7 +311,7 @@ class Client:
|
|||
params: dict,
|
||||
|
||||
method: str = 'get',
|
||||
venue: str|None = None, # if None use `.mkt_mode` state
|
||||
venue: str | None = None, # if None use `.mkt_mode` state
|
||||
signed: bool = False,
|
||||
allow_testnet: bool = False,
|
||||
|
||||
|
|
@ -268,9 +322,8 @@ class Client:
|
|||
- /fapi/v3/ USD-M FUTURES, or
|
||||
- /api/v3/ SPOT/MARGIN
|
||||
|
||||
account/market endpoint request depending on either passed in
|
||||
`venue: str` or the current setting `.mkt_mode: str` setting,
|
||||
default `'spot'`.
|
||||
account/market endpoint request depending on either passed in `venue: str`
|
||||
or the current setting `.mkt_mode: str` setting, default `'spot'`.
|
||||
|
||||
|
||||
Docs per venue API:
|
||||
|
|
@ -299,6 +352,9 @@ class Client:
|
|||
venue=venue_key,
|
||||
)
|
||||
|
||||
sesh: asks.Session
|
||||
path: str
|
||||
|
||||
# Check if we're configured to route order requests to the
|
||||
# venue equivalent's testnet.
|
||||
use_testnet: bool = False
|
||||
|
|
@ -323,12 +379,11 @@ class Client:
|
|||
# ctl machinery B)
|
||||
venue_key += '_testnet'
|
||||
|
||||
client: httpx.AsyncClient
|
||||
path: str
|
||||
client, path = self.venue_sesh[venue_key]
|
||||
meth: Callable = getattr(client, method)
|
||||
sesh, path = self.venue_sesh[venue_key]
|
||||
|
||||
meth: Callable = getattr(sesh, method)
|
||||
resp = await meth(
|
||||
url=path + endpoint,
|
||||
path=path + endpoint,
|
||||
params=params,
|
||||
timeout=float('inf'),
|
||||
)
|
||||
|
|
@ -370,20 +425,7 @@ class Client:
|
|||
item['filters'] = filters
|
||||
|
||||
pair_type: Type = PAIRTYPES[venue]
|
||||
try:
|
||||
pair: Pair = pair_type(**item)
|
||||
except Exception as e:
|
||||
e.add_note(
|
||||
f'\n'
|
||||
f'New or removed field we need to codify!\n'
|
||||
f'pair-type: {pair_type!r}\n'
|
||||
f'\n'
|
||||
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
|
||||
f'Check out their API docs here:\n'
|
||||
f'\n'
|
||||
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
|
||||
)
|
||||
raise
|
||||
pair: Pair = pair_type(**item)
|
||||
pair_table[pair.symbol.upper()] = pair
|
||||
|
||||
# update an additional top-level-cross-venue-table
|
||||
|
|
@ -478,9 +520,7 @@ class Client:
|
|||
|
||||
'''
|
||||
pair_table: dict[str, Pair] = self._venue2pairs[
|
||||
venue
|
||||
or
|
||||
self.mkt_mode
|
||||
venue or self.mkt_mode
|
||||
]
|
||||
if (
|
||||
expiry
|
||||
|
|
@ -499,9 +539,9 @@ class Client:
|
|||
venues: list[str] = [venue]
|
||||
|
||||
# batch per-venue download of all exchange infos
|
||||
async with trio.open_nursery() as tn:
|
||||
async with trio.open_nursery() as rn:
|
||||
for ven in venues:
|
||||
tn.start_soon(
|
||||
rn.start_soon(
|
||||
self._cache_pairs,
|
||||
ven,
|
||||
)
|
||||
|
|
@ -509,7 +549,7 @@ class Client:
|
|||
if sym:
|
||||
return pair_table[sym]
|
||||
else:
|
||||
return self._pairs
|
||||
self._pairs
|
||||
|
||||
async def get_assets(
|
||||
self,
|
||||
|
|
@ -554,32 +594,20 @@ class Client:
|
|||
|
||||
) -> dict[str, Any]:
|
||||
|
||||
fq_pairs: dict[str, Pair] = await self.exch_info()
|
||||
fq_pairs: dict = await self.exch_info()
|
||||
|
||||
# TODO: cache this list like we were in
|
||||
# `open_symbol_search()`?
|
||||
# keys: list[str] = list(fq_pairs)
|
||||
|
||||
return match_from_pairs(
|
||||
pairs=fq_pairs,
|
||||
query=pattern.upper(),
|
||||
matches = fuzzy.extractBests(
|
||||
pattern,
|
||||
fq_pairs,
|
||||
score_cutoff=50,
|
||||
)
|
||||
|
||||
def pair2venuekey(
|
||||
self,
|
||||
pair: Pair,
|
||||
) -> str:
|
||||
return {
|
||||
'USDTM': 'usdtm_futes',
|
||||
'SPOT': 'spot',
|
||||
# 'COINM': 'coin_futes',
|
||||
# ^-TODO-^ bc someone might want it..?
|
||||
}[pair.venue]
|
||||
# repack in dict form
|
||||
return {item[0]['symbol']: item[0]
|
||||
for item in matches}
|
||||
|
||||
async def bars(
|
||||
self,
|
||||
mkt: MktPair,
|
||||
symbol: str,
|
||||
|
||||
start_dt: datetime | None = None,
|
||||
end_dt: datetime | None = None,
|
||||
|
|
@ -609,20 +637,16 @@ class Client:
|
|||
start_time = binance_timestamp(start_dt)
|
||||
end_time = binance_timestamp(end_dt)
|
||||
|
||||
bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
|
||||
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
||||
bars = await self._api(
|
||||
'klines',
|
||||
params={
|
||||
# NOTE: always query using their native symbology!
|
||||
'symbol': mkt.bs_mktid.upper(),
|
||||
'symbol': symbol.upper(),
|
||||
'interval': '1m',
|
||||
'startTime': start_time,
|
||||
'endTime': end_time,
|
||||
'limit': limit
|
||||
},
|
||||
venue=self.pair2venuekey(bs_pair),
|
||||
allow_testnet=False,
|
||||
)
|
||||
new_bars: list[tuple] = []
|
||||
|
|
@ -939,148 +963,17 @@ class Client:
|
|||
await self.close_listen_key(key)
|
||||
|
||||
|
||||
_venue_urls: dict[str, str] = {
|
||||
'spot': (
|
||||
_spot_url,
|
||||
'/api/v3/',
|
||||
),
|
||||
'spot_testnet': (
|
||||
_testnet_spot_url,
|
||||
'/fapi/v1/'
|
||||
),
|
||||
# margin and extended spot endpoints session.
|
||||
# TODO: did this ever get implemented fully?
|
||||
# 'margin': (
|
||||
# _spot_url,
|
||||
# '/sapi/v1/'
|
||||
# ),
|
||||
|
||||
'usdtm_futes': (
|
||||
_futes_url,
|
||||
'/fapi/v1/',
|
||||
),
|
||||
|
||||
'usdtm_futes_testnet': (
|
||||
_testnet_futes_url,
|
||||
'/fapi/v1/',
|
||||
),
|
||||
|
||||
# TODO: for anyone who actually needs it ;P
|
||||
# 'coin_futes': ()
|
||||
}
|
||||
|
||||
|
||||
def init_api_keys(
|
||||
client: Client,
|
||||
conf: dict[str, Any],
|
||||
) -> None:
|
||||
'''
|
||||
Set up per-venue API keys each http client according to the user's
|
||||
`brokers.conf`.
|
||||
|
||||
For ex, to use spot-testnet and live usdt futures APIs:
|
||||
|
||||
```toml
|
||||
[binance]
|
||||
# spot test net
|
||||
spot.use_testnet = true
|
||||
spot.api_key = '<spot_api_key_from_binance_account>'
|
||||
spot.api_secret = '<spot_api_key_password>'
|
||||
|
||||
# futes live
|
||||
futes.use_testnet = false
|
||||
accounts.usdtm = 'futes'
|
||||
futes.api_key = '<futes_api_key_from_binance>'
|
||||
futes.api_secret = '<futes_api_key_password>''
|
||||
|
||||
# if uncommented will use the built-in paper engine and not
|
||||
# connect to `binance` API servers for order ctl.
|
||||
# accounts.paper = 'paper'
|
||||
```
|
||||
|
||||
'''
|
||||
for key, subconf in conf.items():
|
||||
if api_key := subconf.get('api_key', ''):
|
||||
venue_keys: list[str] = client.confkey2venuekeys[key]
|
||||
|
||||
venue_key: str
|
||||
client: httpx.AsyncClient
|
||||
for venue_key in venue_keys:
|
||||
client, _ = client.venue_sesh[venue_key]
|
||||
|
||||
api_key_header: dict = {
|
||||
# taken from official:
|
||||
# https://github.com/binance/binance-futures-connector-python/blob/main/binance/api.py#L47
|
||||
"Content-Type": "application/json;charset=utf-8",
|
||||
|
||||
# TODO: prolly should just always query and copy
|
||||
# in the real latest ver?
|
||||
"User-Agent": "binance-connector/6.1.6smbz6",
|
||||
"X-MBX-APIKEY": api_key,
|
||||
}
|
||||
client.headers.update(api_key_header)
|
||||
|
||||
# if `.use_tesnet = true` in the config then
|
||||
# also add headers for the testnet session which
|
||||
# will be used for all order control
|
||||
if subconf.get('use_testnet', False):
|
||||
testnet_sesh, _ = client.venue_sesh[
|
||||
venue_key + '_testnet'
|
||||
]
|
||||
testnet_sesh.headers.update(api_key_header)
|
||||
|
||||
|
||||
@acm
|
||||
async def get_client(
|
||||
mkt_mode: MarketType = 'spot',
|
||||
) -> Client:
|
||||
'''
|
||||
Construct an single `piker` client which composes multiple underlying venue
|
||||
specific API clients both for live and test networks.
|
||||
async def get_client() -> Client:
|
||||
|
||||
'''
|
||||
venue_sessions: dict[
|
||||
str, # venue key
|
||||
tuple[httpx.AsyncClient, str] # session, eps path
|
||||
] = {}
|
||||
async with AsyncExitStack() as client_stack:
|
||||
for name, (base_url, path) in _venue_urls.items():
|
||||
api: httpx.AsyncClient = await client_stack.enter_async_context(
|
||||
httpx.AsyncClient(
|
||||
base_url=base_url,
|
||||
# headers={},
|
||||
client = Client()
|
||||
await client.exch_info()
|
||||
log.info(
|
||||
f'{client} in {client.mkt_mode} mode: caching exchange infos..\n'
|
||||
'Cached multi-market pairs:\n'
|
||||
f'spot: {len(client._spot_pairs)}\n'
|
||||
f'usdtm_futes: {len(client._ufutes_pairs)}\n'
|
||||
f'Total: {len(client._pairs)}\n'
|
||||
)
|
||||
|
||||
# TODO: is there a way to numerate this?
|
||||
# https://www.python-httpx.org/advanced/clients/#why-use-a-client
|
||||
# connections=4
|
||||
)
|
||||
)
|
||||
venue_sessions[name] = (
|
||||
api,
|
||||
path,
|
||||
)
|
||||
|
||||
conf: dict[str, Any] = get_config()
|
||||
# for creating API keys see,
|
||||
# https://www.binance.com/en/support/faq/how-to-create-api-keys-on-binance-360002502072
|
||||
client = Client(
|
||||
venue_sessions=venue_sessions,
|
||||
conf=conf,
|
||||
mkt_mode=mkt_mode,
|
||||
)
|
||||
init_api_keys(
|
||||
client=client,
|
||||
conf=conf,
|
||||
)
|
||||
fq_pairs: dict[str, Pair] = await client.exch_info()
|
||||
assert fq_pairs
|
||||
log.info(
|
||||
f'Loaded multi-venue `Client` in mkt_mode={client.mkt_mode!r}\n\n'
|
||||
f'Symbology Summary:\n'
|
||||
f'------ - ------\n'
|
||||
f'spot: {len(client._spot_pairs)}\n'
|
||||
f'usdtm_futes: {len(client._ufutes_pairs)}\n'
|
||||
'------ - ------\n'
|
||||
f'total: {len(client._pairs)}\n'
|
||||
)
|
||||
yield client
|
||||
yield client
|
||||
|
|
|
|||
|
|
@ -37,9 +37,8 @@ import trio
|
|||
from piker.accounting import (
|
||||
Asset,
|
||||
)
|
||||
from piker.log import (
|
||||
from piker.brokers._util import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from piker.data._web_bs import (
|
||||
open_autorecon_ws,
|
||||
|
|
@ -70,9 +69,7 @@ from .venues import (
|
|||
)
|
||||
from .api import Client
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
log = get_logger('piker.brokers.binance')
|
||||
|
||||
|
||||
# Fee schedule template, mostly for paper engine fees modelling.
|
||||
|
|
@ -248,16 +245,9 @@ async def handle_order_requests(
|
|||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
|
||||
# enable piker.clearing console log for *this* `brokerd` subactor
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# TODO: how do we set this from the EMS such that
|
||||
# positions are loaded from the correct venue on the user
|
||||
# stream at startup? (that is in an attempt to support both
|
||||
|
|
@ -274,20 +264,15 @@ async def open_trade_dialog(
|
|||
# do a open_symcache() call.. though maybe we can hide
|
||||
# this in a new async version of open_account()?
|
||||
async with open_cached_client('binance') as client:
|
||||
subconf: dict|None = client.conf.get(venue_name)
|
||||
subconf: dict = client.conf[venue_name]
|
||||
use_testnet = subconf.get('use_testnet', False)
|
||||
|
||||
# XXX: if no futes.api_key or spot.api_key has been set we
|
||||
# always fall back to the paper engine!
|
||||
if (
|
||||
not subconf
|
||||
or
|
||||
not subconf.get('api_key')
|
||||
):
|
||||
if not subconf.get('api_key'):
|
||||
await ctx.started('paper')
|
||||
return
|
||||
|
||||
use_testnet: bool = subconf.get('use_testnet', False)
|
||||
|
||||
async with (
|
||||
open_cached_client('binance') as client,
|
||||
):
|
||||
|
|
@ -450,7 +435,6 @@ async def open_trade_dialog(
|
|||
# - ledger: TransactionLedger
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
ctx.open_stream() as ems_stream,
|
||||
):
|
||||
|
|
|
|||
|
|
@ -42,12 +42,12 @@ from trio_typing import TaskStatus
|
|||
from pendulum import (
|
||||
from_timestamp,
|
||||
)
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
import numpy as np
|
||||
import tractor
|
||||
|
||||
from piker.brokers import (
|
||||
open_cached_client,
|
||||
NoData,
|
||||
)
|
||||
from piker._cacheables import (
|
||||
async_lifo_cache,
|
||||
|
|
@ -64,9 +64,9 @@ from piker.data._web_bs import (
|
|||
open_autorecon_ws,
|
||||
NoBsWs,
|
||||
)
|
||||
from piker.log import get_logger
|
||||
from piker.brokers._util import (
|
||||
DataUnavailable,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
from .api import (
|
||||
|
|
@ -78,7 +78,7 @@ from .venues import (
|
|||
get_api_eps,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
log = get_logger('piker.brokers.binance')
|
||||
|
||||
|
||||
class L1(Struct):
|
||||
|
|
@ -94,26 +94,22 @@ class L1(Struct):
|
|||
|
||||
|
||||
# validation type
|
||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Aggregate-Trade-Streams#response-example
|
||||
class AggTrade(Struct, frozen=True):
|
||||
e: str # Event type
|
||||
E: int # Event time
|
||||
s: str # Symbol
|
||||
a: int # Aggregate trade ID
|
||||
p: float # Price
|
||||
q: float # Quantity with all the market trades
|
||||
q: float # Quantity
|
||||
f: int # First trade ID
|
||||
l: int # noqa Last trade ID
|
||||
T: int # Trade time
|
||||
m: bool # Is the buyer the market maker?
|
||||
M: bool|None = None # Ignore
|
||||
nq: float|None = None # Normal quantity without the trades involving RPI orders
|
||||
# ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||
M: bool | None = None # Ignore
|
||||
|
||||
|
||||
async def stream_messages(
|
||||
ws: NoBsWs,
|
||||
|
||||
) -> AsyncGenerator[NoBsWs, dict]:
|
||||
|
||||
# TODO: match syntax here!
|
||||
|
|
@ -224,8 +220,6 @@ def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
|||
}
|
||||
|
||||
|
||||
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
||||
# code?!
|
||||
@acm
|
||||
async def open_history_client(
|
||||
mkt: MktPair,
|
||||
|
|
@ -237,8 +231,8 @@ async def open_history_client(
|
|||
|
||||
async def get_ohlc(
|
||||
timeframe: float,
|
||||
end_dt: datetime|None = None,
|
||||
start_dt: datetime|None = None,
|
||||
end_dt: datetime | None = None,
|
||||
start_dt: datetime | None = None,
|
||||
|
||||
) -> tuple[
|
||||
np.ndarray,
|
||||
|
|
@ -258,36 +252,24 @@ async def open_history_client(
|
|||
else:
|
||||
client.mkt_mode = 'spot'
|
||||
|
||||
array: np.ndarray = await client.bars(
|
||||
mkt=mkt,
|
||||
# NOTE: always query using their native symbology!
|
||||
mktid: str = mkt.bs_mktid
|
||||
array = await client.bars(
|
||||
mktid,
|
||||
start_dt=start_dt,
|
||||
end_dt=end_dt,
|
||||
)
|
||||
if array.size == 0:
|
||||
raise NoData(
|
||||
f'No frame for {start_dt} -> {end_dt}\n'
|
||||
)
|
||||
|
||||
times = array['time']
|
||||
if not times.any():
|
||||
raise ValueError(
|
||||
'Bad frame with null-times?\n\n'
|
||||
f'{times}'
|
||||
)
|
||||
|
||||
# XXX, debug any case where the latest 1m bar we get is
|
||||
# already another "sample's-step-old"..
|
||||
if end_dt is None:
|
||||
inow: int = round(time.time())
|
||||
if (
|
||||
_time_step := (inow - times[-1])
|
||||
>
|
||||
timeframe * 2
|
||||
):
|
||||
if (
|
||||
end_dt is None
|
||||
):
|
||||
inow = round(time.time())
|
||||
if (inow - times[-1]) > 60:
|
||||
await tractor.pause()
|
||||
|
||||
start_dt = from_timestamp(times[0])
|
||||
end_dt = from_timestamp(times[-1])
|
||||
|
||||
return array, start_dt, end_dt
|
||||
|
||||
yield get_ohlc, {'erlangs': 3, 'rate': 3}
|
||||
|
|
@ -297,7 +279,7 @@ async def open_history_client(
|
|||
async def get_mkt_info(
|
||||
fqme: str,
|
||||
|
||||
) -> tuple[MktPair, Pair]|None:
|
||||
) -> tuple[MktPair, Pair] | None:
|
||||
|
||||
# uppercase since kraken bs_mktid is always upper
|
||||
if 'binance' not in fqme.lower():
|
||||
|
|
@ -374,7 +356,7 @@ async def get_mkt_info(
|
|||
if 'futes' in mkt_mode:
|
||||
assert isinstance(pair, FutesPair)
|
||||
|
||||
dst: Asset|None = assets.get(pair.bs_dst_asset)
|
||||
dst: Asset | None = assets.get(pair.bs_dst_asset)
|
||||
if (
|
||||
not dst
|
||||
# TODO: a known asset DNE list?
|
||||
|
|
@ -433,7 +415,7 @@ async def subscribe(
|
|||
# might get ack from ws server, or maybe some
|
||||
# other msg still in transit..
|
||||
res = await ws.recv_msg()
|
||||
subid: str|None = res.get('id')
|
||||
subid: str | None = res.get('id')
|
||||
if subid:
|
||||
assert res['id'] == subid
|
||||
|
||||
|
|
@ -457,6 +439,7 @@ async def subscribe(
|
|||
|
||||
|
||||
async def stream_quotes(
|
||||
|
||||
send_chan: trio.abc.SendChannel,
|
||||
symbols: list[str],
|
||||
feed_is_live: trio.Event,
|
||||
|
|
@ -468,14 +451,11 @@ async def stream_quotes(
|
|||
) -> None:
|
||||
|
||||
async with (
|
||||
tractor.trionics.maybe_raise_from_masking_exc(),
|
||||
send_chan as send_chan,
|
||||
open_cached_client('binance') as client,
|
||||
):
|
||||
init_msgs: list[FeedInit] = []
|
||||
for sym in symbols:
|
||||
mkt: MktPair
|
||||
pair: Pair
|
||||
mkt, pair = await get_mkt_info(sym)
|
||||
|
||||
# build out init msgs according to latest spec
|
||||
|
|
@ -524,6 +504,7 @@ async def stream_quotes(
|
|||
|
||||
# start streaming
|
||||
async for typ, quote in msg_gen:
|
||||
|
||||
# period = time.time() - last
|
||||
# hz = 1/period if period else float('inf')
|
||||
# if hz > 60:
|
||||
|
|
@ -552,15 +533,14 @@ async def open_symbol_search(
|
|||
|
||||
pattern: str
|
||||
async for pattern in stream:
|
||||
# NOTE: pattern fuzzy-matching is done within
|
||||
# the methd impl.
|
||||
pairs: dict[str, Pair] = await client.search_symbols(
|
||||
matches = fuzzy.extractBests(
|
||||
pattern,
|
||||
client._pairs,
|
||||
score_cutoff=50,
|
||||
)
|
||||
|
||||
# repack in fqme-keyed table
|
||||
byfqme: dict[str, Pair] = {}
|
||||
for pair in pairs.values():
|
||||
byfqme[pair.bs_fqme] = pair
|
||||
|
||||
await stream.send(byfqme)
|
||||
# repack in dict form
|
||||
await stream.send({
|
||||
item[0].bs_fqme: item[0]
|
||||
for item in matches
|
||||
})
|
||||
|
|
|
|||
|
|
@ -97,16 +97,6 @@ class Pair(Struct, frozen=True, kw_only=True):
|
|||
baseAsset: str
|
||||
baseAssetPrecision: int
|
||||
|
||||
permissionSets: list[list[str]]
|
||||
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs#2025-08-26
|
||||
# will become non-optional 2025-08-28?
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs#future-changes
|
||||
pegInstructionsAllowed: bool = False
|
||||
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs#2025-12-02
|
||||
opoAllowed: bool = False
|
||||
|
||||
filters: dict[
|
||||
str,
|
||||
str | int | float,
|
||||
|
|
@ -147,17 +137,11 @@ class SpotPair(Pair, frozen=True):
|
|||
quoteOrderQtyMarketAllowed: bool
|
||||
isSpotTradingAllowed: bool
|
||||
isMarginTradingAllowed: bool
|
||||
otoAllowed: bool
|
||||
|
||||
defaultSelfTradePreventionMode: str
|
||||
allowedSelfTradePreventionModes: list[str]
|
||||
permissions: list[str]
|
||||
|
||||
# can the paint botz creat liq gaps even easier on this asset?
|
||||
# Bp
|
||||
# https://developers.binance.com/docs/binance-spot-api-docs/faqs/order_amend_keep_priority
|
||||
amendAllowed: bool
|
||||
|
||||
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
||||
ns_path: str = 'piker.brokers.binance:SpotPair'
|
||||
|
||||
|
|
@ -195,6 +179,7 @@ class FutesPair(Pair):
|
|||
quoteAsset: str # 'USDT',
|
||||
quotePrecision: int # 8,
|
||||
requiredMarginPercent: float # '5.0000',
|
||||
settlePlan: int # 0,
|
||||
timeInForce: list[str] # ['GTC', 'IOC', 'FOK', 'GTX'],
|
||||
triggerProtect: float # '0.0500',
|
||||
underlyingSubType: list[str] # ['PoW'],
|
||||
|
|
@ -209,45 +194,6 @@ class FutesPair(Pair):
|
|||
def quoteAssetPrecision(self) -> int:
|
||||
return self.quotePrecision
|
||||
|
||||
@property
|
||||
def expiry(self) -> str:
|
||||
symbol: str = self.symbol
|
||||
contype: str = self.contractType
|
||||
match contype:
|
||||
case (
|
||||
'CURRENT_QUARTER'
|
||||
| 'CURRENT_QUARTER DELIVERING'
|
||||
| 'NEXT_QUARTER' # su madre binance..
|
||||
):
|
||||
pair, _, expiry = symbol.partition('_')
|
||||
assert pair == self.pair # sanity
|
||||
return f'{expiry}'
|
||||
|
||||
case (
|
||||
'PERPETUAL'
|
||||
| 'TRADIFI_PERPETUAL'
|
||||
):
|
||||
return 'PERP'
|
||||
|
||||
case '':
|
||||
subtype: list[str] = self.underlyingSubType
|
||||
if not subtype:
|
||||
if self.status == 'PENDING_TRADING':
|
||||
return 'PENDING'
|
||||
|
||||
match subtype:
|
||||
case ['DEFI']:
|
||||
return 'PERP'
|
||||
|
||||
# wow, just wow you binance guys suck..
|
||||
if self.status == 'PENDING_TRADING':
|
||||
return 'PENDING'
|
||||
|
||||
# XXX: yeah no clue then..
|
||||
raise ValueError(
|
||||
f'Bad .expiry token match: {contype} for {symbol}'
|
||||
)
|
||||
|
||||
@property
|
||||
def venue(self) -> str:
|
||||
symbol: str = self.symbol
|
||||
|
|
@ -255,54 +201,37 @@ class FutesPair(Pair):
|
|||
margin: str = self.marginAsset
|
||||
|
||||
match ctype:
|
||||
case (
|
||||
'PERPETUAL'
|
||||
| 'TRADIFI_PERPETUAL'
|
||||
):
|
||||
return f'{margin}M'
|
||||
case 'PERPETUAL':
|
||||
return f'{margin}M.PERP'
|
||||
|
||||
case (
|
||||
'CURRENT_QUARTER'
|
||||
| 'CURRENT_QUARTER DELIVERING'
|
||||
| 'NEXT_QUARTER' # su madre binance..
|
||||
):
|
||||
case 'CURRENT_QUARTER':
|
||||
_, _, expiry = symbol.partition('_')
|
||||
return f'{margin}M'
|
||||
return f'{margin}M.{expiry}'
|
||||
|
||||
case '':
|
||||
subtype: list[str] = self.underlyingSubType
|
||||
if not subtype:
|
||||
if self.status == 'PENDING_TRADING':
|
||||
return f'{margin}M'
|
||||
return f'{margin}M.PENDING'
|
||||
|
||||
match subtype:
|
||||
case (
|
||||
['DEFI']
|
||||
| ['USDC']
|
||||
):
|
||||
return f'{subtype[0]}'
|
||||
case ['DEFI']:
|
||||
return f'{subtype[0]}.PERP'
|
||||
|
||||
# XXX: yeah no clue then..
|
||||
raise ValueError(
|
||||
f'Bad .venue token match: {ctype}'
|
||||
)
|
||||
return 'WTF.PWNED.BBQ'
|
||||
|
||||
@property
|
||||
def bs_fqme(self) -> str:
|
||||
symbol: str = self.symbol
|
||||
ctype: str = self.contractType
|
||||
venue: str = self.venue
|
||||
pair: str = self.pair
|
||||
|
||||
match ctype:
|
||||
case (
|
||||
'CURRENT_QUARTER'
|
||||
| 'NEXT_QUARTER' # su madre binance..
|
||||
):
|
||||
pair, _, expiry = symbol.partition('_')
|
||||
assert pair == self.pair
|
||||
case 'CURRENT_QUARTER':
|
||||
symbol, _, expiry = symbol.partition('_')
|
||||
|
||||
return f'{pair}.{venue}.{self.expiry}'
|
||||
return f'{symbol}.{venue}'
|
||||
|
||||
@property
|
||||
def bs_src_asset(self) -> str:
|
||||
|
|
|
|||
|
|
@ -27,12 +27,14 @@ import click
|
|||
import trio
|
||||
import tractor
|
||||
|
||||
from piker.cli import cli
|
||||
from piker import watchlists as wl
|
||||
from piker.log import (
|
||||
from ..cli import cli
|
||||
from .. import watchlists as wl
|
||||
from ..log import (
|
||||
colorize_json,
|
||||
)
|
||||
from ._util import (
|
||||
log,
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ..service import (
|
||||
maybe_spawn_brokerd,
|
||||
|
|
@ -43,15 +45,12 @@ from ..brokers import (
|
|||
get_brokermod,
|
||||
data,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
DEFAULT_BROKER = 'binance'
|
||||
|
||||
_config_dir = click.get_app_dir('piker')
|
||||
_watchlists_data_path = os.path.join(_config_dir, 'watchlists.json')
|
||||
|
||||
|
||||
OK = '\033[92m'
|
||||
WARNING = '\033[93m'
|
||||
FAIL = '\033[91m'
|
||||
|
|
@ -346,10 +345,7 @@ def contracts(ctx, loglevel, broker, symbol, ids):
|
|||
|
||||
'''
|
||||
brokermod = get_brokermod(broker)
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(loglevel)
|
||||
|
||||
contracts = trio.run(partial(core.contracts, brokermod, symbol))
|
||||
if not ids:
|
||||
|
|
@ -458,56 +454,37 @@ def mkt_info(
|
|||
|
||||
@cli.command()
|
||||
@click.argument('pattern', required=True)
|
||||
# TODO: move this to top level click/typer context for all subs
|
||||
@click.option(
|
||||
'--pdb',
|
||||
is_flag=True,
|
||||
help='Enable tractor debug mode',
|
||||
)
|
||||
@click.pass_obj
|
||||
def search(
|
||||
config: dict,
|
||||
pattern: str,
|
||||
pdb: bool,
|
||||
):
|
||||
def search(config, pattern):
|
||||
'''
|
||||
Search for symbols from broker backend(s).
|
||||
|
||||
'''
|
||||
# global opts
|
||||
brokermods: list[ModuleType] = list(config['brokermods'].values())
|
||||
|
||||
# TODO: this is coming from the `search --pdb` NOT from
|
||||
# the `piker --pdb` XD ..
|
||||
# -[ ] pull from the parent click ctx's values..dumdum
|
||||
# assert pdb
|
||||
loglevel: str = config['loglevel']
|
||||
brokermods = list(config['brokermods'].values())
|
||||
|
||||
# define tractor entrypoint
|
||||
async def main(func):
|
||||
|
||||
async with maybe_open_pikerd(
|
||||
loglevel=loglevel,
|
||||
debug_mode=pdb,
|
||||
loglevel=config['loglevel'],
|
||||
):
|
||||
return await func()
|
||||
|
||||
from piker.toolz import open_crash_handler
|
||||
with open_crash_handler():
|
||||
quotes = trio.run(
|
||||
main,
|
||||
partial(
|
||||
core.symbol_search,
|
||||
brokermods,
|
||||
pattern,
|
||||
loglevel=loglevel,
|
||||
),
|
||||
)
|
||||
quotes = trio.run(
|
||||
main,
|
||||
partial(
|
||||
core.symbol_search,
|
||||
brokermods,
|
||||
pattern,
|
||||
),
|
||||
)
|
||||
|
||||
if not quotes:
|
||||
log.error(f"No matches could be found for {pattern}?")
|
||||
return
|
||||
if not quotes:
|
||||
log.error(f"No matches could be found for {pattern}?")
|
||||
return
|
||||
|
||||
click.echo(colorize_json(quotes))
|
||||
click.echo(colorize_json(quotes))
|
||||
|
||||
|
||||
@cli.command()
|
||||
|
|
@ -516,11 +493,9 @@ def search(
|
|||
@click.option('--delete', '-d', flag_value=True, help='Delete section')
|
||||
@click.pass_obj
|
||||
def brokercfg(config, section, value, delete):
|
||||
'''
|
||||
If invoked with no arguments, open an editor to edit broker
|
||||
configs file or get / update an individual section.
|
||||
|
||||
'''
|
||||
"""If invoked with no arguments, open an editor to edit broker configs file
|
||||
or get / update an individual section.
|
||||
"""
|
||||
from .. import config
|
||||
|
||||
if section:
|
||||
|
|
|
|||
|
|
@ -22,26 +22,20 @@ routines should be primitive data types where possible.
|
|||
"""
|
||||
import inspect
|
||||
from types import ModuleType
|
||||
from typing import (
|
||||
Any,
|
||||
)
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
import trio
|
||||
|
||||
from piker.log import get_logger
|
||||
from ._util import log
|
||||
from . import get_brokermod
|
||||
from ..service import maybe_spawn_brokerd
|
||||
from . import open_cached_client
|
||||
from ..accounting import MktPair
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
async def api(brokername: str, methname: str, **kwargs) -> dict:
|
||||
'''
|
||||
Make (proxy through) a broker API call by name and return its result.
|
||||
|
||||
'''
|
||||
"""Make (proxy through) a broker API call by name and return its result.
|
||||
"""
|
||||
brokermod = get_brokermod(brokername)
|
||||
async with brokermod.get_client() as client:
|
||||
meth = getattr(client, methname, None)
|
||||
|
|
@ -68,14 +62,10 @@ async def api(brokername: str, methname: str, **kwargs) -> dict:
|
|||
|
||||
async def stocks_quote(
|
||||
brokermod: ModuleType,
|
||||
tickers: list[str]
|
||||
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
'''
|
||||
Return a `dict` of snapshot quotes for the provided input
|
||||
`tickers`: a `list` of fqmes.
|
||||
|
||||
'''
|
||||
tickers: List[str]
|
||||
) -> Dict[str, Dict[str, Any]]:
|
||||
"""Return quotes dict for ``tickers``.
|
||||
"""
|
||||
async with brokermod.get_client() as client:
|
||||
return await client.quote(tickers)
|
||||
|
||||
|
|
@ -84,15 +74,13 @@ async def stocks_quote(
|
|||
async def option_chain(
|
||||
brokermod: ModuleType,
|
||||
symbol: str,
|
||||
date: str|None = None,
|
||||
) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
'''
|
||||
Return option chain for ``symbol`` for ``date``.
|
||||
date: Optional[str] = None,
|
||||
) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
"""Return option chain for ``symbol`` for ``date``.
|
||||
|
||||
By default all expiries are returned. If ``date`` is provided
|
||||
then contract quotes for that single expiry are returned.
|
||||
|
||||
'''
|
||||
"""
|
||||
async with brokermod.get_client() as client:
|
||||
if date:
|
||||
id = int((await client.tickers2ids([symbol]))[symbol])
|
||||
|
|
@ -110,7 +98,7 @@ async def option_chain(
|
|||
# async def contracts(
|
||||
# brokermod: ModuleType,
|
||||
# symbol: str,
|
||||
# ) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
# ) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
# """Return option contracts (all expiries) for ``symbol``.
|
||||
# """
|
||||
# async with brokermod.get_client() as client:
|
||||
|
|
@ -122,24 +110,15 @@ async def bars(
|
|||
brokermod: ModuleType,
|
||||
symbol: str,
|
||||
**kwargs,
|
||||
) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
'''
|
||||
Return option contracts (all expiries) for ``symbol``.
|
||||
|
||||
'''
|
||||
) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
"""Return option contracts (all expiries) for ``symbol``.
|
||||
"""
|
||||
async with brokermod.get_client() as client:
|
||||
return await client.bars(symbol, **kwargs)
|
||||
|
||||
|
||||
async def search_w_brokerd(
|
||||
name: str,
|
||||
pattern: str,
|
||||
) -> dict:
|
||||
async def search_w_brokerd(name: str, pattern: str) -> dict:
|
||||
|
||||
# TODO: WHY NOT WORK!?!
|
||||
# when we `step` through the next block?
|
||||
# import tractor
|
||||
# await tractor.pause()
|
||||
async with open_cached_client(name) as client:
|
||||
|
||||
# TODO: support multiple asset type concurrent searches.
|
||||
|
|
@ -149,15 +128,14 @@ async def search_w_brokerd(
|
|||
async def symbol_search(
|
||||
brokermods: list[ModuleType],
|
||||
pattern: str,
|
||||
loglevel: str = 'warning',
|
||||
**kwargs,
|
||||
|
||||
) -> dict[str, dict[str, dict[str, Any]]]:
|
||||
) -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
'''
|
||||
Return symbol info from broker.
|
||||
|
||||
'''
|
||||
results: list[str] = []
|
||||
results = []
|
||||
|
||||
async def search_backend(
|
||||
brokermod: ModuleType
|
||||
|
|
@ -165,21 +143,9 @@ async def symbol_search(
|
|||
|
||||
brokername: str = mod.name
|
||||
|
||||
# TODO: figure this the FUCK OUT
|
||||
# -> ok so obvi in the root actor any async task that's
|
||||
# spawned outside the main tractor-root-actor task needs to
|
||||
# call this..
|
||||
# await tractor.devx._debug.maybe_init_greenback()
|
||||
# tractor.pause_from_sync()
|
||||
|
||||
async with maybe_spawn_brokerd(
|
||||
mod.name,
|
||||
infect_asyncio=getattr(
|
||||
mod,
|
||||
'_infect_asyncio',
|
||||
False,
|
||||
),
|
||||
loglevel=loglevel
|
||||
infect_asyncio=getattr(mod, '_infect_asyncio', False),
|
||||
) as portal:
|
||||
|
||||
results.append((
|
||||
|
|
@ -192,6 +158,7 @@ async def symbol_search(
|
|||
))
|
||||
|
||||
async with trio.open_nursery() as n:
|
||||
|
||||
for mod in brokermods:
|
||||
n.start_soon(search_backend, mod.name)
|
||||
|
||||
|
|
@ -201,13 +168,11 @@ async def symbol_search(
|
|||
async def mkt_info(
|
||||
brokermod: ModuleType,
|
||||
fqme: str,
|
||||
|
||||
**kwargs,
|
||||
|
||||
) -> MktPair:
|
||||
'''
|
||||
Return the `piker.accounting.MktPair` info struct from a given
|
||||
backend broker tradable src/dst asset pair.
|
||||
Return MktPair info from broker including src and dst assets.
|
||||
|
||||
'''
|
||||
async with open_cached_client(brokermod.name) as client:
|
||||
|
|
|
|||
|
|
@ -41,15 +41,12 @@ import tractor
|
|||
from tractor.experimental import msgpub
|
||||
from async_generator import asynccontextmanager
|
||||
|
||||
from piker.log import(
|
||||
get_logger,
|
||||
from ._util import (
|
||||
log,
|
||||
get_console_log,
|
||||
)
|
||||
from . import get_brokermod
|
||||
|
||||
log = get_logger(
|
||||
name='piker.brokers.binance',
|
||||
)
|
||||
|
||||
async def wait_for_network(
|
||||
net_func: Callable,
|
||||
|
|
@ -246,10 +243,7 @@ async def start_quote_stream(
|
|||
|
||||
'''
|
||||
# XXX: why do we need this again?
|
||||
get_console_log(
|
||||
level=tractor.current_actor().loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(tractor.current_actor().loglevel)
|
||||
|
||||
# pull global vars from local actor
|
||||
symbols = list(symbols)
|
||||
|
|
|
|||
|
|
@ -31,15 +31,14 @@ from typing import (
|
|||
Callable,
|
||||
)
|
||||
|
||||
from pendulum import now
|
||||
import pendulum
|
||||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
from rapidfuzz import process as fuzzy
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
import numpy as np
|
||||
from tractor.trionics import (
|
||||
broadcast_receiver,
|
||||
maybe_open_context
|
||||
collapse_eg,
|
||||
)
|
||||
from tractor import to_asyncio
|
||||
# XXX WOOPS XD
|
||||
|
|
@ -53,11 +52,8 @@ from cryptofeed.defines import (
|
|||
)
|
||||
from cryptofeed.symbols import Symbol
|
||||
|
||||
from piker.data import (
|
||||
def_iohlcv_fields,
|
||||
match_from_pairs,
|
||||
Struct,
|
||||
)
|
||||
from piker.data.types import Struct
|
||||
from piker.data import def_iohlcv_fields
|
||||
from piker.data._web_bs import (
|
||||
open_jsonrpc_session
|
||||
)
|
||||
|
|
@ -83,7 +79,7 @@ _testnet_ws_url = 'wss://test.deribit.com/ws/api/v2'
|
|||
class JSONRPCResult(Struct):
|
||||
jsonrpc: str = '2.0'
|
||||
id: int
|
||||
result: Optional[list[dict]] = None
|
||||
result: Optional[dict] = None
|
||||
error: Optional[dict] = None
|
||||
usIn: int
|
||||
usOut: int
|
||||
|
|
@ -293,29 +289,24 @@ class Client:
|
|||
currency: str = 'btc', # BTC, ETH, SOL, USDC
|
||||
kind: str = 'option',
|
||||
expired: bool = False
|
||||
) -> dict[str, Any]:
|
||||
"""Get symbol info for the exchange.
|
||||
|
||||
) -> dict[str, dict]:
|
||||
'''
|
||||
Get symbol infos.
|
||||
|
||||
'''
|
||||
"""
|
||||
if self._pairs:
|
||||
return self._pairs
|
||||
|
||||
# will retrieve all symbols by default
|
||||
params: dict[str, str] = {
|
||||
params = {
|
||||
'currency': currency.upper(),
|
||||
'kind': kind,
|
||||
'expired': str(expired).lower()
|
||||
}
|
||||
|
||||
resp: JSONRPCResult = await self.json_rpc(
|
||||
'public/get_instruments',
|
||||
params,
|
||||
)
|
||||
# convert to symbol-keyed table
|
||||
results: list[dict] | None = resp.result
|
||||
instruments: dict[str, dict] = {
|
||||
resp = await self.json_rpc('public/get_instruments', params)
|
||||
results = resp.result
|
||||
|
||||
instruments = {
|
||||
item['instrument_name'].lower(): item
|
||||
for item in results
|
||||
}
|
||||
|
|
@ -328,7 +319,6 @@ class Client:
|
|||
async def cache_symbols(
|
||||
self,
|
||||
) -> dict:
|
||||
|
||||
if not self._pairs:
|
||||
self._pairs = await self.symbol_info()
|
||||
|
||||
|
|
@ -339,23 +329,17 @@ class Client:
|
|||
pattern: str,
|
||||
limit: int = 30,
|
||||
) -> dict[str, Any]:
|
||||
'''
|
||||
Fuzzy search symbology set for pairs matching `pattern`.
|
||||
data = await self.symbol_info()
|
||||
|
||||
'''
|
||||
pairs: dict[str, Any] = await self.symbol_info()
|
||||
matches: dict[str, Pair] = match_from_pairs(
|
||||
pairs=pairs,
|
||||
query=pattern.upper(),
|
||||
matches = fuzzy.extractBests(
|
||||
pattern,
|
||||
data,
|
||||
score_cutoff=35,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
# repack in name-keyed table
|
||||
return {
|
||||
pair['instrument_name'].lower(): pair
|
||||
for pair in matches.values()
|
||||
}
|
||||
# repack in dict form
|
||||
return {item[0]['instrument_name'].lower(): item[0]
|
||||
for item in matches}
|
||||
|
||||
async def bars(
|
||||
self,
|
||||
|
|
@ -433,7 +417,6 @@ async def get_client(
|
|||
) -> Client:
|
||||
|
||||
async with (
|
||||
collapse_eg(),
|
||||
trio.open_nursery() as n,
|
||||
open_jsonrpc_session(
|
||||
_testnet_ws_url, dtype=JSONRPCResult) as json_rpc
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ import time
|
|||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
import pendulum
|
||||
from rapidfuzz import process as fuzzy
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
import numpy as np
|
||||
import tractor
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
--------------
|
||||
more or less the "everything broker" for traditional and international
|
||||
markets. they are the "go to" provider for automatic retail trading
|
||||
and we interface to their APIs using the `ib_async` project.
|
||||
and we interface to their APIs using the `ib_insync` project.
|
||||
|
||||
status
|
||||
******
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ Sub-modules within break into the core functionalities:
|
|||
- ``broker.py`` part for orders / trading endpoints
|
||||
- ``feed.py`` for real-time data feed endpoints
|
||||
- ``api.py`` for the core API machinery which is ``trio``-ized
|
||||
wrapping around `ib_async`.
|
||||
wrapping around ``ib_insync``.
|
||||
|
||||
"""
|
||||
from .api import (
|
||||
|
|
|
|||
|
|
@ -111,7 +111,7 @@ def load_flex_trades(
|
|||
|
||||
) -> dict[str, Any]:
|
||||
|
||||
from ib_async import flexreport, util
|
||||
from ib_insync import flexreport, util
|
||||
|
||||
conf = get_config()
|
||||
|
||||
|
|
@ -154,7 +154,8 @@ def load_flex_trades(
|
|||
trade_entries,
|
||||
)
|
||||
|
||||
ledger_dict: dict|None
|
||||
ledger_dict: dict | None = None
|
||||
|
||||
for acctid in trades_by_account:
|
||||
trades_by_id = trades_by_account[acctid]
|
||||
|
||||
|
|
|
|||
|
|
@ -20,12 +20,6 @@ runnable script-programs.
|
|||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
import asyncio
|
||||
from datetime import ( # noqa
|
||||
datetime,
|
||||
date,
|
||||
tzinfo as TzInfo,
|
||||
)
|
||||
from functools import partial
|
||||
from typing import (
|
||||
Literal,
|
||||
|
|
@ -35,13 +29,13 @@ import subprocess
|
|||
|
||||
import tractor
|
||||
|
||||
from piker.log import get_logger
|
||||
from piker.brokers._util import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .api import Client
|
||||
import i3ipc
|
||||
from ib_insync import IB
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
log = get_logger('piker.brokers.ib')
|
||||
|
||||
_reset_tech: Literal[
|
||||
'vnc',
|
||||
|
|
@ -54,39 +48,8 @@ _reset_tech: Literal[
|
|||
] = 'vnc'
|
||||
|
||||
|
||||
no_setup_msg:str = (
|
||||
'No data reset hack test setup for {vnc_sockaddr}!\n'
|
||||
'See config setup tips @\n'
|
||||
'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
|
||||
)
|
||||
|
||||
|
||||
def try_xdo_manual(
|
||||
client: Client,
|
||||
):
|
||||
'''
|
||||
Do the "manual" `xdo`-based screen switch + click
|
||||
combo since apparently the `asyncvnc` client ain't workin..
|
||||
|
||||
Note this is only meant as a backup method for Xorg users,
|
||||
ideally you can use a real vnc client and the `vnc_click_hack()`
|
||||
impl!
|
||||
|
||||
'''
|
||||
global _reset_tech
|
||||
try:
|
||||
i3ipc_xdotool_manual_click_hack()
|
||||
_reset_tech = 'i3ipc_xdotool'
|
||||
return True
|
||||
except OSError:
|
||||
vnc_sockaddr: str = client.conf.vnc_addrs
|
||||
log.exception(
|
||||
no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
async def data_reset_hack(
|
||||
# vnc_host: str,
|
||||
client: Client,
|
||||
reset_type: Literal['data', 'connection'],
|
||||
|
||||
|
|
@ -118,138 +81,80 @@ async def data_reset_hack(
|
|||
that need to be wrangle.
|
||||
|
||||
'''
|
||||
ib_client: IB = client.ib
|
||||
|
||||
# look up any user defined vnc socket address mapped from
|
||||
# a particular API socket port.
|
||||
vnc_addrs: tuple[str]|None = client.conf.get('vnc_addrs')
|
||||
if not vnc_addrs:
|
||||
log.warning(
|
||||
no_setup_msg.format(vnc_sockaddr=client.conf)
|
||||
+
|
||||
'REQUIRES A `vnc_addrs: array` ENTRY'
|
||||
)
|
||||
api_port: str = str(ib_client.client.port)
|
||||
vnc_host: str
|
||||
vnc_port: int
|
||||
vnc_host, vnc_port = client.conf['vnc_addrs'].get(
|
||||
api_port,
|
||||
('localhost', 3003)
|
||||
)
|
||||
|
||||
no_setup_msg:str = (
|
||||
f'No data reset hack test setup for {vnc_host}!\n'
|
||||
'See setup @\n'
|
||||
'https://github.com/pikers/piker/tree/master/piker/brokers/ib'
|
||||
)
|
||||
global _reset_tech
|
||||
|
||||
match _reset_tech:
|
||||
case 'vnc':
|
||||
try:
|
||||
await tractor.to_asyncio.run_task(
|
||||
partial(
|
||||
vnc_click_hack,
|
||||
client=client,
|
||||
host=vnc_host,
|
||||
port=vnc_port,
|
||||
)
|
||||
)
|
||||
except (
|
||||
OSError, # no VNC server avail..
|
||||
PermissionError, # asyncvnc pw fail..
|
||||
) as _vnc_err:
|
||||
vnc_err = _vnc_err
|
||||
except OSError:
|
||||
if vnc_host != 'localhost':
|
||||
log.warning(no_setup_msg)
|
||||
return False
|
||||
|
||||
try:
|
||||
import i3ipc # noqa (since a deps dynamic check)
|
||||
except ModuleNotFoundError:
|
||||
log.warning(
|
||||
no_setup_msg.format(vnc_sockaddr=client.conf)
|
||||
)
|
||||
log.warning(no_setup_msg)
|
||||
return False
|
||||
|
||||
# XXX, Xorg only workaround..
|
||||
# TODO? remove now that we have `pyvnc`?
|
||||
# if vnc_host not in {
|
||||
# 'localhost',
|
||||
# '127.0.0.1',
|
||||
# }:
|
||||
# focussed, matches = i3ipc_fin_wins_titled()
|
||||
# if not matches:
|
||||
# log.warning(
|
||||
# no_setup_msg.format(vnc_sockaddr=vnc_sockaddr)
|
||||
# )
|
||||
# return False
|
||||
# else:
|
||||
# try_xdo_manual(vnc_sockaddr)
|
||||
|
||||
# localhost but no vnc-client or it borked..
|
||||
else:
|
||||
log.error(
|
||||
'VNC CLICK HACK FAILE with,\n'
|
||||
f'{vnc_err!r}\n'
|
||||
)
|
||||
|
||||
# breakpoint()
|
||||
# try_xdo_manual(client)
|
||||
try:
|
||||
i3ipc_xdotool_manual_click_hack()
|
||||
_reset_tech = 'i3ipc_xdotool'
|
||||
return True
|
||||
except OSError:
|
||||
log.exception(no_setup_msg)
|
||||
return False
|
||||
|
||||
case 'i3ipc_xdotool':
|
||||
try_xdo_manual(client)
|
||||
# i3ipc_xdotool_manual_click_hack()
|
||||
i3ipc_xdotool_manual_click_hack()
|
||||
|
||||
case _ as tech:
|
||||
raise RuntimeError(
|
||||
f'{tech!r} is not supported for reset tech!?'
|
||||
)
|
||||
raise RuntimeError(f'{tech} is not supported for reset tech!?')
|
||||
|
||||
# we don't really need the ``xdotool`` approach any more B)
|
||||
return True
|
||||
|
||||
|
||||
async def vnc_click_hack(
|
||||
client: Client,
|
||||
reset_type: str = 'data',
|
||||
pw: str|None = None,
|
||||
|
||||
host: str,
|
||||
port: int,
|
||||
reset_type: str = 'data'
|
||||
) -> None:
|
||||
'''
|
||||
Reset the data or network connection for the VNC attached
|
||||
ib-gateway using a (magic) keybinding combo.
|
||||
|
||||
A vnc-server password can be set either by an input `pw` param or
|
||||
set in the client's config with the latter loaded from the user's
|
||||
`brokers.toml` in a vnc-addrs-port-mapping section,
|
||||
|
||||
.. code:: toml
|
||||
|
||||
[ib.vnc_addrs]
|
||||
4002 = {host = 'localhost', port = 5900, pw = 'doggy'}
|
||||
ib gateway using magic combos.
|
||||
|
||||
'''
|
||||
api_port: str = str(client.ib.client.port)
|
||||
conf: dict = client.conf
|
||||
vnc_addrs: dict[int, tuple] = conf.get('vnc_addrs')
|
||||
if not vnc_addrs:
|
||||
return None
|
||||
|
||||
addr_entry: dict|tuple = vnc_addrs.get(
|
||||
api_port,
|
||||
('localhost', 5900) # a typical default
|
||||
)
|
||||
if pw is None:
|
||||
match addr_entry:
|
||||
case (
|
||||
host,
|
||||
port,
|
||||
):
|
||||
pass
|
||||
|
||||
case {
|
||||
'host': host,
|
||||
'port': port,
|
||||
'pw': pw
|
||||
}:
|
||||
pass
|
||||
|
||||
case _:
|
||||
raise ValueError(
|
||||
f'Invalid `ib.vnc_addrs` entry ?\n'
|
||||
f'{addr_entry!r}\n'
|
||||
)
|
||||
try:
|
||||
from pyvnc import (
|
||||
AsyncVNCClient,
|
||||
VNCConfig,
|
||||
Point,
|
||||
MOUSE_BUTTON_LEFT,
|
||||
)
|
||||
import asyncvnc
|
||||
except ModuleNotFoundError:
|
||||
log.warning(
|
||||
"In order to leverage `piker`'s built-in data reset hacks, install "
|
||||
"the `pyvnc` project: https://github.com/regulad/pyvnc.git"
|
||||
"the `asyncvnc` project: https://github.com/barneygale/asyncvnc"
|
||||
)
|
||||
return
|
||||
|
||||
|
|
@ -260,105 +165,24 @@ async def vnc_click_hack(
|
|||
'connection': 'r'
|
||||
}[reset_type]
|
||||
|
||||
with tractor.devx.open_crash_handler(
|
||||
ignore={TimeoutError,},
|
||||
):
|
||||
client = await AsyncVNCClient.connect(
|
||||
VNCConfig(
|
||||
host=host,
|
||||
port=port,
|
||||
password=pw,
|
||||
)
|
||||
async with asyncvnc.connect(
|
||||
host,
|
||||
port=port,
|
||||
|
||||
# TODO: doesn't work see:
|
||||
# https://github.com/barneygale/asyncvnc/issues/7
|
||||
# password='ibcansmbz',
|
||||
|
||||
) as client:
|
||||
|
||||
# move to middle of screen
|
||||
# 640x1800
|
||||
client.mouse.move(
|
||||
x=500,
|
||||
y=500,
|
||||
)
|
||||
async with client:
|
||||
# move to middle of screen
|
||||
# 640x1800
|
||||
await client.move(
|
||||
Point(
|
||||
500, # x from left
|
||||
400, # y from top
|
||||
)
|
||||
)
|
||||
# in case a prior dialog win is open/active.
|
||||
await client.press('ISO_Enter')
|
||||
|
||||
# ensure the ib-gw window is active
|
||||
await client.click(MOUSE_BUTTON_LEFT)
|
||||
|
||||
# send the hotkeys combo B)
|
||||
await client.press(
|
||||
'Ctrl',
|
||||
'Alt',
|
||||
key,
|
||||
) # NOTE, keys are stacked
|
||||
|
||||
# XXX, sometimes a dialog asking if you want to "simulate
|
||||
# a reset" will show, in which case we want to select
|
||||
# "Yes" (by tabbing) and then hit enter.
|
||||
iters: int = 1
|
||||
delay: float = 0.3
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
for i in range(iters):
|
||||
log.info(f'Sending TAB {i}')
|
||||
await client.press('Tab')
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
for i in range(iters):
|
||||
log.info(f'Sending ENTER {i}')
|
||||
await client.press('KP_Enter')
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
|
||||
def i3ipc_fin_wins_titled(
|
||||
titles: list[str] = [
|
||||
'Interactive Brokers', # tws running in i3
|
||||
'IB Gateway', # gw running in i3
|
||||
# 'IB', # gw running in i3 (newer version?)
|
||||
|
||||
# !TODO, remote vnc instance
|
||||
# -[ ] something in title (or other Con-props) that indicates
|
||||
# this is explicitly for ibrk sw?
|
||||
# |_[ ] !can use modden spawn eventually!
|
||||
'TigerVNC',
|
||||
# 'vncviewer', # the terminal..
|
||||
],
|
||||
) -> tuple[
|
||||
i3ipc.Con, # orig focussed win
|
||||
list[tuple[str, i3ipc.Con]], # matching wins by title
|
||||
]:
|
||||
'''
|
||||
Attempt to find a local-DE window titled with an entry in
|
||||
`titles`.
|
||||
|
||||
If found deliver the current focussed window and all matching
|
||||
`i3ipc.Con`s in a list.
|
||||
|
||||
'''
|
||||
import i3ipc
|
||||
ipc = i3ipc.Connection()
|
||||
|
||||
# TODO: might be worth offering some kinda api for grabbing
|
||||
# the window id from the pid?
|
||||
# https://stackoverflow.com/a/2250879
|
||||
tree = ipc.get_tree()
|
||||
focussed: i3ipc.Con = tree.find_focused()
|
||||
|
||||
matches: list[i3ipc.Con] = []
|
||||
for name in titles:
|
||||
results = tree.find_titled(name)
|
||||
print(f'results for {name}: {results}')
|
||||
if results:
|
||||
con = results[0]
|
||||
matches.append((
|
||||
name,
|
||||
con,
|
||||
))
|
||||
|
||||
return (
|
||||
focussed,
|
||||
matches,
|
||||
)
|
||||
client.mouse.click()
|
||||
client.keyboard.press('Ctrl', 'Alt', key) # keys are stacked
|
||||
|
||||
|
||||
def i3ipc_xdotool_manual_click_hack() -> None:
|
||||
|
|
@ -366,55 +190,67 @@ def i3ipc_xdotool_manual_click_hack() -> None:
|
|||
Do the data reset hack but expecting a local X-window using `xdotool`.
|
||||
|
||||
'''
|
||||
focussed, matches = i3ipc_fin_wins_titled()
|
||||
try:
|
||||
orig_win_id = focussed.window
|
||||
except AttributeError:
|
||||
# XXX if .window cucks we prolly aren't intending to
|
||||
# use this and/or just woke up from suspend..
|
||||
log.exception('xdotool invalid usage ya ??\n')
|
||||
return
|
||||
import i3ipc
|
||||
i3 = i3ipc.Connection()
|
||||
|
||||
# TODO: might be worth offering some kinda api for grabbing
|
||||
# the window id from the pid?
|
||||
# https://stackoverflow.com/a/2250879
|
||||
t = i3.get_tree()
|
||||
|
||||
orig_win_id = t.find_focused().window
|
||||
|
||||
# for tws
|
||||
win_names: list[str] = [
|
||||
'Interactive Brokers', # tws running in i3
|
||||
'IB Gateway', # gw running in i3
|
||||
# 'IB', # gw running in i3 (newer version?)
|
||||
]
|
||||
|
||||
try:
|
||||
for name, con in matches:
|
||||
print(f'Resetting data feed for {name}')
|
||||
win_id = str(con.window)
|
||||
w, h = con.rect.width, con.rect.height
|
||||
for name in win_names:
|
||||
results = t.find_titled(name)
|
||||
print(f'results for {name}: {results}')
|
||||
if results:
|
||||
con = results[0]
|
||||
print(f'Resetting data feed for {name}')
|
||||
win_id = str(con.window)
|
||||
w, h = con.rect.width, con.rect.height
|
||||
|
||||
# TODO: seems to be a few libs for python but not sure
|
||||
# if they support all the sub commands we need, order of
|
||||
# most recent commit history:
|
||||
# https://github.com/rr-/pyxdotool
|
||||
# https://github.com/ShaneHutter/pyxdotool
|
||||
# https://github.com/cphyc/pyxdotool
|
||||
# TODO: seems to be a few libs for python but not sure
|
||||
# if they support all the sub commands we need, order of
|
||||
# most recent commit history:
|
||||
# https://github.com/rr-/pyxdotool
|
||||
# https://github.com/ShaneHutter/pyxdotool
|
||||
# https://github.com/cphyc/pyxdotool
|
||||
|
||||
# TODO: only run the reconnect (2nd) kc on a detected
|
||||
# disconnect?
|
||||
for key_combo, timeout in [
|
||||
# only required if we need a connection reset.
|
||||
# ('ctrl+alt+r', 12),
|
||||
# data feed reset.
|
||||
('ctrl+alt+f', 6)
|
||||
]:
|
||||
subprocess.call([
|
||||
'xdotool',
|
||||
'windowactivate', '--sync', win_id,
|
||||
# TODO: only run the reconnect (2nd) kc on a detected
|
||||
# disconnect?
|
||||
for key_combo, timeout in [
|
||||
# only required if we need a connection reset.
|
||||
# ('ctrl+alt+r', 12),
|
||||
# data feed reset.
|
||||
('ctrl+alt+f', 6)
|
||||
]:
|
||||
subprocess.call([
|
||||
'xdotool',
|
||||
'windowactivate', '--sync', win_id,
|
||||
|
||||
# move mouse to bottom left of window (where
|
||||
# there should be nothing to click).
|
||||
'mousemove_relative', '--sync', str(w-4), str(h-4),
|
||||
# move mouse to bottom left of window (where
|
||||
# there should be nothing to click).
|
||||
'mousemove_relative', '--sync', str(w-4), str(h-4),
|
||||
|
||||
# NOTE: we may need to stick a `--retry 3` in here..
|
||||
'click', '--window', win_id,
|
||||
'--repeat', '3', '1',
|
||||
# NOTE: we may need to stick a `--retry 3` in here..
|
||||
'click', '--window', win_id,
|
||||
'--repeat', '3', '1',
|
||||
|
||||
# hackzorzes
|
||||
'key', key_combo,
|
||||
],
|
||||
timeout=timeout,
|
||||
)
|
||||
# hackzorzes
|
||||
'key', key_combo,
|
||||
],
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
# re-activate and focus original window
|
||||
# re-activate and focus original window
|
||||
subprocess.call([
|
||||
'xdotool',
|
||||
'windowactivate', '--sync', str(orig_win_id),
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -20,7 +20,7 @@ Order and trades endpoints for use with ``piker``'s EMS.
|
|||
"""
|
||||
from __future__ import annotations
|
||||
from contextlib import ExitStack
|
||||
# from collections import ChainMap
|
||||
from collections import ChainMap
|
||||
from functools import partial
|
||||
from pprint import pformat
|
||||
import time
|
||||
|
|
@ -34,15 +34,14 @@ import trio
|
|||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from tractor.to_asyncio import LinkedTaskChannel
|
||||
from tractor import trionics
|
||||
from ib_async.contract import (
|
||||
from ib_insync.contract import (
|
||||
Contract,
|
||||
)
|
||||
from ib_async.order import (
|
||||
from ib_insync.order import (
|
||||
Trade,
|
||||
OrderStatus,
|
||||
)
|
||||
from ib_async.objects import (
|
||||
from ib_insync.objects import (
|
||||
Fill,
|
||||
Execution,
|
||||
CommissionReport,
|
||||
|
|
@ -50,10 +49,6 @@ from ib_async.objects import (
|
|||
)
|
||||
|
||||
from piker import config
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.accounting import (
|
||||
Position,
|
||||
|
|
@ -81,6 +76,7 @@ from piker.clearing._messages import (
|
|||
BrokerdFill,
|
||||
BrokerdError,
|
||||
)
|
||||
from ._util import log
|
||||
from .api import (
|
||||
_accounts2clients,
|
||||
get_config,
|
||||
|
|
@ -98,10 +94,6 @@ from .ledger import (
|
|||
update_ledger_from_api_trades,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
|
||||
def pack_position(
|
||||
pos: IbPosition,
|
||||
|
|
@ -124,11 +116,7 @@ def pack_position(
|
|||
symbol=fqme,
|
||||
currency=con.currency,
|
||||
size=float(pos.position),
|
||||
avg_price=(
|
||||
float(pos.avgCost)
|
||||
/
|
||||
float(con.multiplier or 1.0)
|
||||
),
|
||||
avg_price=float(pos.avgCost) / float(con.multiplier or 1.0),
|
||||
),
|
||||
)
|
||||
|
||||
|
|
@ -181,7 +169,7 @@ async def handle_order_requests(
|
|||
# validate
|
||||
order = BrokerdOrder(**request_msg)
|
||||
|
||||
# XXX: by default 0 tells ``ib_async`` methods that
|
||||
# XXX: by default 0 tells ``ib_insync`` methods that
|
||||
# there is no existing order so ask the client to create
|
||||
# a new one (which it seems to do by allocating an int
|
||||
# counter - collision prone..)
|
||||
|
|
@ -237,7 +225,7 @@ async def recv_trade_updates(
|
|||
) -> None:
|
||||
'''
|
||||
Receive and relay order control and positioning related events
|
||||
from `ib_async`, pack as tuples and push over mem-chan to our
|
||||
from `ib_insync`, pack as tuples and push over mem-chan to our
|
||||
trio relay task for processing and relay to EMS.
|
||||
|
||||
'''
|
||||
|
|
@ -303,7 +291,7 @@ async def recv_trade_updates(
|
|||
# much more then a few more pnl fields..
|
||||
# 'updatePortfolioEvent',
|
||||
|
||||
# XXX: these all seem to be weird ib_async internal
|
||||
# XXX: these all seem to be weird ib_insync internal
|
||||
# events that we probably don't care that much about
|
||||
# given the internal design is wonky af..
|
||||
# 'newOrderEvent',
|
||||
|
|
@ -369,10 +357,6 @@ async def update_and_audit_pos_msg(
|
|||
size=ibpos.position,
|
||||
|
||||
avg_price=pikerpos.ppu,
|
||||
|
||||
# XXX ensures matching even if multiple venue-names
|
||||
# in `.bs_fqme`, likely from txn records..
|
||||
bs_mktid=mkt.bs_mktid,
|
||||
)
|
||||
|
||||
ibfmtmsg: str = pformat(ibpos._asdict())
|
||||
|
|
@ -423,7 +407,7 @@ async def update_and_audit_pos_msg(
|
|||
|
||||
# TODO: make this a "propaganda" log level?
|
||||
if ibpos.avgCost != msg.avg_price:
|
||||
log.debug(
|
||||
log.warning(
|
||||
f'IB "FIFO" avg price for {msg.symbol} is DIFF:\n'
|
||||
f'ib: {ibfmtmsg}\n'
|
||||
'---------------------------\n'
|
||||
|
|
@ -441,8 +425,7 @@ async def aggr_open_orders(
|
|||
|
||||
) -> None:
|
||||
'''
|
||||
Collect all open orders from client and fill in `order_msgs:
|
||||
list`.
|
||||
Collect all open orders from client and fill in `order_msgs: list`.
|
||||
|
||||
'''
|
||||
trades: list[Trade] = client.ib.openTrades()
|
||||
|
|
@ -499,7 +482,7 @@ async def open_trade_event_stream(
|
|||
] = trio.TASK_STATUS_IGNORED,
|
||||
):
|
||||
'''
|
||||
Proxy wrapper for starting trade event stream from ib_async
|
||||
Proxy wrapper for starting trade event stream from ib_insync
|
||||
which spawns an asyncio task that registers an internal closure
|
||||
(`push_tradies()`) which in turn relays trading events through
|
||||
a `tractor.to_asyncio.LinkedTaskChannel` which the parent
|
||||
|
|
@ -543,15 +526,9 @@ class IbAcnt(Struct):
|
|||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# task local msg dialog tracking
|
||||
flows = OrderDialogs()
|
||||
accounts_def = config.load_accounts(['ib'])
|
||||
|
|
@ -569,10 +546,7 @@ async def open_trade_dialog(
|
|||
),
|
||||
|
||||
# TODO: do this as part of `open_account()`!?
|
||||
open_symcache(
|
||||
'ib',
|
||||
only_from_memcache=True,
|
||||
) as symcache,
|
||||
open_symcache('ib', only_from_memcache=True) as symcache,
|
||||
):
|
||||
# Open a trade ledgers stack for appending trade records over
|
||||
# multiple accounts.
|
||||
|
|
@ -580,10 +554,8 @@ async def open_trade_dialog(
|
|||
ledgers: dict[str, TransactionLedger] = {}
|
||||
tables: dict[str, Account] = {}
|
||||
order_msgs: list[Status] = []
|
||||
conf: dict = get_config()
|
||||
accounts_def_inv: bidict[str, str] = bidict(
|
||||
conf['accounts']
|
||||
).inverse
|
||||
conf = get_config()
|
||||
accounts_def_inv: bidict[str, str] = bidict(conf['accounts']).inverse
|
||||
|
||||
with (
|
||||
ExitStack() as lstack,
|
||||
|
|
@ -733,11 +705,7 @@ async def open_trade_dialog(
|
|||
# client-account and build out position msgs to deliver to
|
||||
# EMS.
|
||||
for acctid, acnt in tables.items():
|
||||
active_pps: dict[str, Position]
|
||||
(
|
||||
active_pps,
|
||||
closed_pps,
|
||||
) = acnt.dump_active()
|
||||
active_pps, closed_pps = acnt.dump_active()
|
||||
|
||||
for pps in [active_pps, closed_pps]:
|
||||
piker_pps: list[Position] = list(pps.values())
|
||||
|
|
@ -753,7 +721,6 @@ async def open_trade_dialog(
|
|||
)
|
||||
if ibpos:
|
||||
bs_mktid: str = str(ibpos.contract.conId)
|
||||
|
||||
msg = await update_and_audit_pos_msg(
|
||||
acctid,
|
||||
pikerpos,
|
||||
|
|
@ -771,7 +738,7 @@ async def open_trade_dialog(
|
|||
f'UNEXPECTED POSITION says IB => {msg.symbol}\n'
|
||||
'Maybe they LIQUIDATED YOU or your ledger is wrong?\n'
|
||||
)
|
||||
log.debug(logmsg)
|
||||
log.error(logmsg)
|
||||
|
||||
await ctx.started((
|
||||
all_positions,
|
||||
|
|
@ -780,22 +747,21 @@ async def open_trade_dialog(
|
|||
|
||||
async with (
|
||||
ctx.open_stream() as ems_stream,
|
||||
trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
trio.open_nursery() as n,
|
||||
):
|
||||
# relay existing open orders to ems
|
||||
for msg in order_msgs:
|
||||
await ems_stream.send(msg)
|
||||
|
||||
for client in set(aioclients.values()):
|
||||
trade_event_stream: LinkedTaskChannel = await tn.start(
|
||||
trade_event_stream: LinkedTaskChannel = await n.start(
|
||||
open_trade_event_stream,
|
||||
client,
|
||||
)
|
||||
|
||||
# start order request handler **before** local trades
|
||||
# event loop
|
||||
tn.start_soon(
|
||||
n.start_soon(
|
||||
handle_order_requests,
|
||||
ems_stream,
|
||||
accounts_def,
|
||||
|
|
@ -803,7 +769,7 @@ async def open_trade_dialog(
|
|||
)
|
||||
|
||||
# allocate event relay tasks for each client connection
|
||||
tn.start_soon(
|
||||
n.start_soon(
|
||||
deliver_trade_events,
|
||||
|
||||
trade_event_stream,
|
||||
|
|
@ -880,18 +846,6 @@ async def emit_pp_update(
|
|||
|
||||
# con: Contract = fill.contract
|
||||
|
||||
# provide a backup fqme -> MktPair table in case the
|
||||
# symcache does not (yet) have an entry for the current mkt
|
||||
# txn.
|
||||
backup_table: dict[str, MktPair] = {}
|
||||
for tid, txn in trans.items():
|
||||
fqme: str = txn.fqme
|
||||
if fqme not in ledger.symcache.mktmaps:
|
||||
# bs_mktid: str = txn.bs_mktid
|
||||
backup_table[fqme] = client._cons2mkts[
|
||||
client._contracts[fqme]
|
||||
]
|
||||
|
||||
acnt.update_from_ledger(
|
||||
trans,
|
||||
|
||||
|
|
@ -901,7 +855,7 @@ async def emit_pp_update(
|
|||
|
||||
# TODO: remove this hack by attempting to symcache an
|
||||
# incrementally updated table?
|
||||
_mktmap_table=backup_table,
|
||||
_mktmap_table=client._contracts
|
||||
)
|
||||
|
||||
# re-compute all positions that have changed state.
|
||||
|
|
@ -991,9 +945,6 @@ _statuses: dict[str, str] = {
|
|||
# TODO: see a current ``ib_insync`` issue around this:
|
||||
# https://github.com/erdewit/ib_insync/issues/363
|
||||
'Inactive': 'pending',
|
||||
|
||||
# XXX, uhh wut the heck is this?
|
||||
'ValidationError': 'error',
|
||||
}
|
||||
|
||||
_action_map = {
|
||||
|
|
@ -1066,19 +1017,8 @@ async def deliver_trade_events(
|
|||
# TODO: for some reason we can receive a ``None`` here when the
|
||||
# ib-gw goes down? Not sure exactly how that's happening looking
|
||||
# at the eventkit code above but we should probably handle it...
|
||||
event_name: str
|
||||
item: (
|
||||
Trade
|
||||
|tuple[Trade, Fill]
|
||||
|CommissionReport
|
||||
|IbPosition
|
||||
|dict
|
||||
)
|
||||
async for event_name, item in trade_event_stream:
|
||||
log.info(
|
||||
f'Relaying {event_name!r}:\n'
|
||||
f'{pformat(item)}\n'
|
||||
)
|
||||
log.info(f'Relaying `{event_name}`:\n{pformat(item)}')
|
||||
match event_name:
|
||||
case 'orderStatusEvent':
|
||||
|
||||
|
|
@ -1089,12 +1029,11 @@ async def deliver_trade_events(
|
|||
trade: Trade = item
|
||||
reqid: str = str(trade.order.orderId)
|
||||
status: OrderStatus = trade.orderStatus
|
||||
status_str: str = _statuses.get(
|
||||
status.status,
|
||||
'error',
|
||||
)
|
||||
status_str: str = _statuses[status.status]
|
||||
remaining: float = status.remaining
|
||||
if status_str == 'filled':
|
||||
if (
|
||||
status_str == 'filled'
|
||||
):
|
||||
fill: Fill = trade.fills[-1]
|
||||
execu: Execution = fill.execution
|
||||
|
||||
|
|
@ -1125,12 +1064,6 @@ async def deliver_trade_events(
|
|||
# all units were cleared.
|
||||
status_str = 'closed'
|
||||
|
||||
elif status_str == 'error':
|
||||
log.error(
|
||||
f'IB reported error status for order ??\n'
|
||||
f'{status.status!r}\n'
|
||||
)
|
||||
|
||||
# skip duplicate filled updates - we get the deats
|
||||
# from the execution details event
|
||||
msg = BrokerdStatus(
|
||||
|
|
@ -1238,14 +1171,7 @@ async def deliver_trade_events(
|
|||
pos
|
||||
and fill
|
||||
):
|
||||
now_cr: CommissionReport = fill.commissionReport
|
||||
if (now_cr != cr):
|
||||
log.warning(
|
||||
'UhhHh ib updated the commission report mid-fill..?\n'
|
||||
f'was: {pformat(cr)}\n'
|
||||
f'now: {pformat(now_cr)}\n'
|
||||
)
|
||||
|
||||
assert fill.commissionReport == cr
|
||||
await emit_pp_update(
|
||||
ems_stream,
|
||||
accounts_def,
|
||||
|
|
@ -1291,67 +1217,39 @@ async def deliver_trade_events(
|
|||
case 'error':
|
||||
# NOTE: see impl deats in
|
||||
# `Client.inline_errors()::push_err()`
|
||||
err: dict|str = item
|
||||
err: dict = item
|
||||
|
||||
# std case, never relay errors for non-order-control
|
||||
# related issues.
|
||||
# never relay errors for non-broker related issues
|
||||
# https://interactivebrokers.github.io/tws-api/message_codes.html
|
||||
if isinstance(err, dict):
|
||||
code: int = err['error_code']
|
||||
reason: str = err['reason']
|
||||
reqid: str = str(err['reqid'])
|
||||
|
||||
# XXX, sometimes you'll get just a `str` of the form,
|
||||
# '[code 104] connection failed' or something..
|
||||
elif isinstance(err, str):
|
||||
code_part, _, reason = err.rpartition(']')
|
||||
if code_part:
|
||||
_, _, code = code_part.partition('[code')
|
||||
reqid: str = '<unknown>'
|
||||
|
||||
# "Warning:" msg codes,
|
||||
# https://interactivebrokers.github.io/tws-api/message_codes.html#warning_codes
|
||||
# - 2109: 'Outside Regular Trading Hours'
|
||||
if 'Warning:' in reason:
|
||||
log.warning(
|
||||
f'Order-API-warning: {code!r}\n'
|
||||
f'reqid: {reqid!r}\n'
|
||||
f'\n'
|
||||
f'{pformat(err)}\n'
|
||||
# ^TODO? should we just print the `reason`
|
||||
# not the full `err`-dict?
|
||||
)
|
||||
continue
|
||||
|
||||
# XXX known special (ignore) cases
|
||||
elif code in {
|
||||
200, # uhh.. ni idea
|
||||
code: int = err['error_code']
|
||||
if code in {
|
||||
200, # uhh
|
||||
|
||||
# hist pacing / connectivity
|
||||
162,
|
||||
165,
|
||||
|
||||
# WARNING codes:
|
||||
# https://interactivebrokers.github.io/tws-api/message_codes.html#warning_codes
|
||||
# Attribute 'Outside Regular Trading Hours' is
|
||||
# " 'ignored based on the order type and
|
||||
# destination. PlaceOrder is now ' 'being
|
||||
# processed.',
|
||||
2109,
|
||||
|
||||
# XXX: lol this isn't even documented..
|
||||
# 'No market data during competing live session'
|
||||
1669,
|
||||
}:
|
||||
log.error(
|
||||
f'Order-API-error which is non-cancel-causing ?!\n'
|
||||
f'\n'
|
||||
f'{pformat(err)}\n'
|
||||
)
|
||||
continue
|
||||
|
||||
if err['reqid'] == -1:
|
||||
log.error(
|
||||
f'TWS external order error ??\n'
|
||||
f'{pformat(err)}\n'
|
||||
)
|
||||
reqid: str = str(err['reqid'])
|
||||
reason: str = err['reason']
|
||||
|
||||
flow: dict = dict(
|
||||
flows.get(reqid)
|
||||
or {}
|
||||
)
|
||||
if err['reqid'] == -1:
|
||||
log.error(f'TWS external order error:\n{pformat(err)}')
|
||||
|
||||
flow: ChainMap = flows.get(reqid)
|
||||
|
||||
# TODO: we don't want to relay data feed / lookup errors
|
||||
# so we need some further filtering logic here..
|
||||
|
|
@ -1362,7 +1260,7 @@ async def deliver_trade_events(
|
|||
reason=reason,
|
||||
broker_details={
|
||||
'name': 'ib',
|
||||
'flow': flow,
|
||||
'flow': dict(flow),
|
||||
},
|
||||
)
|
||||
flows.add_msg(reqid, err_msg.to_dict())
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -31,20 +31,14 @@ from typing import (
|
|||
)
|
||||
|
||||
from bidict import bidict
|
||||
from pendulum import (
|
||||
DateTime,
|
||||
parse,
|
||||
from_timestamp,
|
||||
)
|
||||
from ib_async import (
|
||||
import pendulum
|
||||
from ib_insync.objects import (
|
||||
Contract,
|
||||
Commodity,
|
||||
Fill,
|
||||
Execution,
|
||||
CommissionReport,
|
||||
)
|
||||
|
||||
from piker.log import get_logger
|
||||
from piker.types import Struct
|
||||
from piker.data import (
|
||||
SymbologyCache,
|
||||
|
|
@ -58,6 +52,7 @@ from piker.accounting import (
|
|||
iter_by_dt,
|
||||
)
|
||||
from ._flex_reports import parse_flex_dt
|
||||
from ._util import log
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .api import (
|
||||
|
|
@ -65,19 +60,15 @@ if TYPE_CHECKING:
|
|||
MethodProxy,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
tx_sort: Callable = partial(
|
||||
iter_by_dt,
|
||||
parsers={
|
||||
'dateTime': parse_flex_dt,
|
||||
'datetime': parse,
|
||||
|
||||
# XXX: for some some fucking 2022 and
|
||||
# back options records.. f@#$ me..
|
||||
'date': parse,
|
||||
'datetime': pendulum.parse,
|
||||
# for some some fucking 2022 and
|
||||
# back options records...fuck me.
|
||||
'date': pendulum.parse,
|
||||
}
|
||||
)
|
||||
|
||||
|
|
@ -97,38 +88,15 @@ def norm_trade(
|
|||
|
||||
conid: int = str(record.get('conId') or record['conid'])
|
||||
bs_mktid: str = str(conid)
|
||||
comms = record.get('commission')
|
||||
if comms is None:
|
||||
comms = -1*record['ibCommission']
|
||||
|
||||
# NOTE: sometimes weird records (like BTTX?)
|
||||
# have no field for this?
|
||||
comms: float = -1 * (
|
||||
record.get('commission')
|
||||
or record.get('ibCommission')
|
||||
or 0
|
||||
)
|
||||
if not comms:
|
||||
log.warning(
|
||||
'No commissions found for record?\n'
|
||||
f'{pformat(record)}\n'
|
||||
)
|
||||
|
||||
price: float = (
|
||||
record.get('price')
|
||||
or record.get('tradePrice')
|
||||
)
|
||||
if price is None:
|
||||
log.warning(
|
||||
'No `price` field found in record?\n'
|
||||
'Skipping normalization..\n'
|
||||
f'{pformat(record)}\n'
|
||||
)
|
||||
return None
|
||||
price = record.get('price') or record['tradePrice']
|
||||
|
||||
# the api doesn't do the -/+ on the quantity for you but flex
|
||||
# records do.. are you fucking serious ib...!?
|
||||
size: float|int = (
|
||||
record.get('quantity')
|
||||
or record['shares']
|
||||
) * {
|
||||
size = record.get('quantity') or record['shares'] * {
|
||||
'BOT': 1,
|
||||
'SLD': -1,
|
||||
}[record['side']]
|
||||
|
|
@ -159,31 +127,26 @@ def norm_trade(
|
|||
# otype = tail[6]
|
||||
# strike = tail[7:]
|
||||
|
||||
log.warning(
|
||||
f'Skipping option contract -> NO SUPPORT YET!\n'
|
||||
f'{symbol}\n'
|
||||
)
|
||||
print(f'skipping opts contract {symbol}')
|
||||
return None
|
||||
|
||||
# timestamping is way different in API records
|
||||
dtstr: str = record.get('datetime')
|
||||
date: str = record.get('date')
|
||||
flex_dtstr: str = record.get('dateTime')
|
||||
dtstr = record.get('datetime')
|
||||
date = record.get('date')
|
||||
flex_dtstr = record.get('dateTime')
|
||||
|
||||
if dtstr or date:
|
||||
dt: DateTime = parse(dtstr or date)
|
||||
dt = pendulum.parse(dtstr or date)
|
||||
|
||||
elif flex_dtstr:
|
||||
# probably a flex record with a wonky non-std timestamp..
|
||||
dt: DateTime = parse_flex_dt(record['dateTime'])
|
||||
dt = parse_flex_dt(record['dateTime'])
|
||||
|
||||
# special handling of symbol extraction from
|
||||
# flex records using some ad-hoc schema parsing.
|
||||
asset_type: str = (
|
||||
record.get('assetCategory')
|
||||
or record.get('secType')
|
||||
or 'STK'
|
||||
)
|
||||
asset_type: str = record.get(
|
||||
'assetCategory'
|
||||
) or record.get('secType', 'STK')
|
||||
|
||||
if (expiry := (
|
||||
record.get('lastTradeDateOrContractMonth')
|
||||
|
|
@ -274,21 +237,6 @@ def norm_trade(
|
|||
name=symbol.lower(),
|
||||
atype='option',
|
||||
tx_tick=Decimal('1'),
|
||||
|
||||
# TODO: we should probably always cast to the
|
||||
# `Contract` instance then dict-serialize that for
|
||||
# the `.info` field!
|
||||
# info=asdict(Option()),
|
||||
)
|
||||
|
||||
case 'CMDTY':
|
||||
from .symbols import _adhoc_symbol_map
|
||||
con_kwargs, _ = _adhoc_symbol_map[symbol.upper()]
|
||||
dst = Asset(
|
||||
name=symbol.lower(),
|
||||
atype='commodity',
|
||||
tx_tick=Decimal('1'),
|
||||
info=asdict(Commodity(**con_kwargs)),
|
||||
)
|
||||
|
||||
# try to build out piker fqme from record.
|
||||
|
|
@ -393,7 +341,6 @@ def norm_trade_records(
|
|||
if txn is None:
|
||||
continue
|
||||
|
||||
# inject txns sorted by datetime
|
||||
insort(
|
||||
records,
|
||||
txn,
|
||||
|
|
@ -442,7 +389,7 @@ def api_trades_to_ledger_entries(
|
|||
txn_dict[attr_name] = val
|
||||
|
||||
tid = str(txn_dict['execId'])
|
||||
dt = from_timestamp(txn_dict['time'])
|
||||
dt = pendulum.from_timestamp(txn_dict['time'])
|
||||
txn_dict['datetime'] = str(dt)
|
||||
acctid = accounts[txn_dict['acctNumber']]
|
||||
|
||||
|
|
|
|||
|
|
@ -23,17 +23,15 @@ from contextlib import (
|
|||
nullcontext,
|
||||
)
|
||||
from decimal import Decimal
|
||||
from functools import partial
|
||||
import time
|
||||
from typing import (
|
||||
Awaitable,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
from rapidfuzz import process as fuzzy
|
||||
import ib_async as ibis
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
import ib_insync as ibis
|
||||
import tractor
|
||||
from tractor.devx.pformat import ppfmt
|
||||
import trio
|
||||
|
||||
from piker.accounting import (
|
||||
|
|
@ -44,7 +42,10 @@ from piker.accounting import (
|
|||
from piker._cacheables import (
|
||||
async_lifo_cache,
|
||||
)
|
||||
from piker.log import get_logger
|
||||
|
||||
from ._util import (
|
||||
log,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .api import (
|
||||
|
|
@ -52,10 +53,6 @@ if TYPE_CHECKING:
|
|||
Client,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
_futes_venues = (
|
||||
'GLOBEX',
|
||||
'NYMEX',
|
||||
|
|
@ -137,7 +134,7 @@ _adhoc_fiat_set = set((
|
|||
|
||||
# manually discovered tick discrepancies,
|
||||
# onl god knows how or why they'd cuck these up..
|
||||
_adhoc_mkt_infos: dict[int|str, dict] = {
|
||||
_adhoc_mkt_infos: dict[int | str, dict] = {
|
||||
'vtgn.nasdaq': {'price_tick': Decimal('0.01')},
|
||||
}
|
||||
|
||||
|
|
@ -168,7 +165,6 @@ _exch_skip_list = {
|
|||
'MEXI', # mexican stocks
|
||||
|
||||
# no idea
|
||||
'NSE',
|
||||
'VALUE',
|
||||
'FUNDSERV',
|
||||
'SWB2',
|
||||
|
|
@ -212,24 +208,20 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
|
|||
break
|
||||
|
||||
ib_client = proxy._aio_ns.ib
|
||||
log.info(
|
||||
f'Using API client for symbol-search\n'
|
||||
f'{ib_client}\n'
|
||||
)
|
||||
log.info(f'Using {ib_client} for symbol search')
|
||||
|
||||
last: float = time.time()
|
||||
last = time.time()
|
||||
async for pattern in stream:
|
||||
log.info(f'received {pattern}')
|
||||
now: float = time.time()
|
||||
now = time.time()
|
||||
|
||||
# TODO? check this is no longer true?
|
||||
# this causes tractor hang...
|
||||
# assert 0
|
||||
|
||||
assert pattern, 'IB can not accept blank search pattern'
|
||||
|
||||
# throttle search requests to no faster then 1Hz
|
||||
diff: float = now - last
|
||||
diff = now - last
|
||||
if diff < 1.0:
|
||||
log.debug('throttle sleeping')
|
||||
await trio.sleep(diff)
|
||||
|
|
@ -240,12 +232,11 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
|
|||
|
||||
if (
|
||||
not pattern
|
||||
or
|
||||
pattern.isspace()
|
||||
or
|
||||
or pattern.isspace()
|
||||
|
||||
# XXX: not sure if this is a bad assumption but it
|
||||
# seems to make search snappier?
|
||||
len(pattern) < 1
|
||||
or len(pattern) < 1
|
||||
):
|
||||
log.warning('empty pattern received, skipping..')
|
||||
|
||||
|
|
@ -258,58 +249,34 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
|
|||
# XXX: this unblocks the far end search task which may
|
||||
# hold up a multi-search nursery block
|
||||
await stream.send({})
|
||||
|
||||
continue
|
||||
|
||||
log.info(
|
||||
f'Searching for FQME with,\n'
|
||||
f'pattern: {pattern!r}\n'
|
||||
)
|
||||
log.info(f'searching for {pattern}')
|
||||
|
||||
last: float = time.time()
|
||||
last = time.time()
|
||||
|
||||
# async batch search using api stocks endpoint and
|
||||
# module defined adhoc symbol set.
|
||||
stock_results: list[dict] = []
|
||||
# async batch search using api stocks endpoint and module
|
||||
# defined adhoc symbol set.
|
||||
stock_results = []
|
||||
|
||||
async def extend_results(
|
||||
# ?TODO, how to type async-fn!?
|
||||
target: Awaitable[list],
|
||||
pattern: str,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
async def stash_results(target: Awaitable[list]):
|
||||
try:
|
||||
results = await target(
|
||||
pattern=pattern,
|
||||
**kwargs,
|
||||
)
|
||||
client_repr: str = proxy._aio_ns.ib.client.__class__.__name__
|
||||
meth_repr: str = target.keywords["meth"]
|
||||
log.info(
|
||||
f'Search query,\n'
|
||||
f'{client_repr}.{meth_repr}(\n'
|
||||
f' pattern={pattern!r}\n'
|
||||
f' **kwargs={kwargs!r},\n'
|
||||
f') = {ppfmt(list(results))}'
|
||||
# XXX ^ just the keys since that's what
|
||||
# shows in UI results table.
|
||||
)
|
||||
results = await target
|
||||
except tractor.trionics.Lagged:
|
||||
log.exception(
|
||||
'IB SYM-SEARCH OVERRUN?!?\n'
|
||||
)
|
||||
print("IB SYM-SEARCH OVERRUN?!?")
|
||||
return
|
||||
|
||||
stock_results.extend(results)
|
||||
|
||||
for _ in range(10):
|
||||
for i in range(10):
|
||||
with trio.move_on_after(3) as cs:
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(
|
||||
partial(
|
||||
extend_results,
|
||||
async with trio.open_nursery() as sn:
|
||||
sn.start_soon(
|
||||
stash_results,
|
||||
proxy.search_symbols(
|
||||
pattern=pattern,
|
||||
target=proxy.search_symbols,
|
||||
upto=10,
|
||||
upto=5,
|
||||
),
|
||||
)
|
||||
|
||||
|
|
@ -321,13 +288,11 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
|
|||
f'Search timeout? {proxy._aio_ns.ib.client}'
|
||||
)
|
||||
continue
|
||||
elif stock_results:
|
||||
else:
|
||||
break
|
||||
# else:
|
||||
# await tractor.pause()
|
||||
|
||||
# # match against our ad-hoc set immediately
|
||||
# adhoc_matches = fuzzy.extract(
|
||||
# adhoc_matches = fuzzy.extractBests(
|
||||
# pattern,
|
||||
# list(_adhoc_futes_set),
|
||||
# score_cutoff=90,
|
||||
|
|
@ -339,10 +304,8 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
|
|||
# adhoc_match_results = {i[0]: {} for i in
|
||||
# adhoc_matches}
|
||||
|
||||
log.debug(
|
||||
f'fuzzy matching stocks {ppfmt(stock_results)}'
|
||||
)
|
||||
stock_matches = fuzzy.extract(
|
||||
log.debug(f'fuzzy matching stocks {stock_results}')
|
||||
stock_matches = fuzzy.extractBests(
|
||||
pattern,
|
||||
stock_results,
|
||||
score_cutoff=50,
|
||||
|
|
@ -355,10 +318,7 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
|
|||
# TODO: we used to deliver contract details
|
||||
# {item[2]: item[0] for item in stock_matches}
|
||||
|
||||
log.debug(
|
||||
f'Sending final matches\n'
|
||||
f'{matches.keys()}'
|
||||
)
|
||||
log.debug(f"sending matches: {matches.keys()}")
|
||||
await stream.send(matches)
|
||||
|
||||
|
||||
|
|
@ -463,9 +423,9 @@ def con2fqme(
|
|||
except KeyError:
|
||||
pass
|
||||
|
||||
suffix: str = con.primaryExchange or con.exchange
|
||||
symbol: str = con.symbol
|
||||
expiry: str = con.lastTradeDateOrContractMonth or ''
|
||||
suffix = con.primaryExchange or con.exchange
|
||||
symbol = con.symbol
|
||||
expiry = con.lastTradeDateOrContractMonth or ''
|
||||
|
||||
match con:
|
||||
case ibis.Option():
|
||||
|
|
@ -520,7 +480,8 @@ def con2fqme(
|
|||
@async_lifo_cache()
|
||||
async def get_mkt_info(
|
||||
fqme: str,
|
||||
proxy: MethodProxy|None = None,
|
||||
|
||||
proxy: MethodProxy | None = None,
|
||||
|
||||
) -> tuple[MktPair, ibis.ContractDetails]:
|
||||
|
||||
|
|
@ -553,28 +514,10 @@ async def get_mkt_info(
|
|||
if atype == 'commodity':
|
||||
venue: str = 'cmdty'
|
||||
else:
|
||||
venue: str = (
|
||||
con.primaryExchange
|
||||
or
|
||||
con.exchange
|
||||
)
|
||||
venue = con.primaryExchange or con.exchange
|
||||
|
||||
price_tick: Decimal = Decimal(str(details.minTick))
|
||||
ib_min_tick_gt_2: Decimal = Decimal('0.01')
|
||||
if (
|
||||
price_tick < ib_min_tick_gt_2
|
||||
):
|
||||
# TODO: we need to add some kinda dynamic rounding sys
|
||||
# to our MktPair i guess?
|
||||
# not sure where the logic should sit, but likely inside
|
||||
# the `.clearing._ems` i suppose...
|
||||
log.warning(
|
||||
'IB seems to disallow a min price tick < 0.01 '
|
||||
'when the price is > 2.0..?\n'
|
||||
f'Decreasing min tick precision for {fqme} to 0.01'
|
||||
)
|
||||
# price_tick = ib_min_tick
|
||||
# await tractor.pause()
|
||||
# price_tick: Decimal = Decimal('0.01')
|
||||
|
||||
if atype == 'stock':
|
||||
# XXX: GRRRR they don't support fractional share sizes for
|
||||
|
|
@ -585,7 +528,7 @@ async def get_mkt_info(
|
|||
size_tick: Decimal = Decimal(
|
||||
str(details.minSize).rstrip('0')
|
||||
)
|
||||
# ?TODO, there is also the Contract.sizeIncrement, bt wtf is it?
|
||||
# |-> TODO: there is also the Contract.sizeIncrement, bt wtf is it?
|
||||
|
||||
# NOTE: this is duplicate from the .broker.norm_trade_records()
|
||||
# routine, we should factor all this parsing somewhere..
|
||||
|
|
|
|||
|
|
@ -1,325 +0,0 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) Tyler Goodlet (in stewardship for pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
(Multi-)venue mgmt helpers.
|
||||
|
||||
IB generally supports all "legacy" trading venues, those mostly owned
|
||||
by ICE and friends.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
from datetime import ( # noqa
|
||||
datetime,
|
||||
date,
|
||||
tzinfo as TzInfo,
|
||||
)
|
||||
from typing import (
|
||||
Iterator,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import exchange_calendars as xcals
|
||||
from pendulum import (
|
||||
now,
|
||||
Duration,
|
||||
Interval,
|
||||
Time,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ib_async import (
|
||||
TradingSession,
|
||||
Contract,
|
||||
ContractDetails,
|
||||
)
|
||||
from exchange_calendars.exchange_calendars import (
|
||||
ExchangeCalendar,
|
||||
)
|
||||
from pandas import (
|
||||
# DatetimeIndex,
|
||||
TimeDelta,
|
||||
Timestamp,
|
||||
)
|
||||
|
||||
|
||||
def has_weekend(
|
||||
period: Interval,
|
||||
) -> bool:
|
||||
'''
|
||||
Predicate to for a period being within
|
||||
days 6->0 (sat->sun).
|
||||
|
||||
'''
|
||||
has_weekend: bool = False
|
||||
for dt in period:
|
||||
if dt.day_of_week in [0, 6]: # 0=Sunday, 6=Saturday
|
||||
has_weekend = True
|
||||
break
|
||||
|
||||
return has_weekend
|
||||
|
||||
|
||||
def has_holiday(
|
||||
con_deats: ContractDetails,
|
||||
period: Interval,
|
||||
) -> bool:
|
||||
'''
|
||||
Using the `exchange_calendars` lib detect if a time-gap `period`
|
||||
is contained in a known "cash hours" closure.
|
||||
|
||||
'''
|
||||
tz: str = con_deats.timeZoneId
|
||||
con: Contract = con_deats.contract
|
||||
exch: str = (
|
||||
con.primaryExchange
|
||||
or
|
||||
con.exchange
|
||||
)
|
||||
|
||||
# XXX, ad-hoc handle any IB exchange which are non-std
|
||||
# via lookup table..
|
||||
std_exch: dict = {
|
||||
'ARCA': 'ARCX',
|
||||
}.get(exch, exch)
|
||||
|
||||
cal: ExchangeCalendar = xcals.get_calendar(std_exch)
|
||||
end: datetime = period.end
|
||||
# _start: datetime = period.start
|
||||
# ?TODO, can rm ya?
|
||||
# => not that useful?
|
||||
# dti: DatetimeIndex = cal.sessions_in_range(
|
||||
# _start.date(),
|
||||
# end.date(),
|
||||
# )
|
||||
prev_close: Timestamp = cal.previous_close(
|
||||
end.date()
|
||||
).tz_convert(tz)
|
||||
prev_open: Timestamp = cal.previous_open(
|
||||
end.date()
|
||||
).tz_convert(tz)
|
||||
# now do relative from prev_ values ^
|
||||
# to get the next open which should match
|
||||
# "contain" the end of the gap.
|
||||
next_open: Timestamp = cal.next_open(
|
||||
prev_open,
|
||||
).tz_convert(tz)
|
||||
next_open: Timestamp = cal.next_open(
|
||||
prev_open,
|
||||
).tz_convert(tz)
|
||||
_next_close: Timestamp = cal.next_close(
|
||||
prev_close
|
||||
).tz_convert(tz)
|
||||
cash_gap: TimeDelta = next_open - prev_close
|
||||
is_holiday_gap = (
|
||||
cash_gap
|
||||
>
|
||||
period
|
||||
)
|
||||
# XXX, debug
|
||||
# breakpoint()
|
||||
return is_holiday_gap
|
||||
|
||||
|
||||
def is_current_time_in_range(
|
||||
sesh: Interval,
|
||||
when: datetime|None = None,
|
||||
) -> bool:
|
||||
'''
|
||||
Check if current time is within the datetime range.
|
||||
|
||||
Use any/the-same timezone as provided by `start_dt.tzinfo` value
|
||||
in the range.
|
||||
|
||||
'''
|
||||
when: datetime = when or now()
|
||||
return when in sesh
|
||||
|
||||
|
||||
def iter_sessions(
|
||||
con_deats: ContractDetails,
|
||||
) -> Iterator[Interval]:
|
||||
'''
|
||||
Yield `pendulum.Interval`s for all
|
||||
`ibas.ContractDetails.tradingSessions() -> TradingSession`s.
|
||||
|
||||
'''
|
||||
sesh: TradingSession
|
||||
for sesh in con_deats.tradingSessions():
|
||||
yield Interval(*sesh)
|
||||
|
||||
|
||||
def sesh_times(
|
||||
con_deats: ContractDetails,
|
||||
) -> tuple[Time, Time]:
|
||||
'''
|
||||
Based on the earliest trading session provided by the IB API,
|
||||
get the (day-agnostic) times for the start/end.
|
||||
|
||||
'''
|
||||
earliest_sesh: Interval = next(iter_sessions(con_deats))
|
||||
return (
|
||||
earliest_sesh.start.time(),
|
||||
earliest_sesh.end.time(),
|
||||
)
|
||||
# ^?TODO, use `.diff()` to get point-in-time-agnostic period?
|
||||
# https://pendulum.eustace.io/docs/#difference
|
||||
|
||||
|
||||
def is_venue_open(
|
||||
con_deats: ContractDetails,
|
||||
when: datetime|Duration|None = None,
|
||||
) -> bool:
|
||||
'''
|
||||
Check if market-venue is open during `when`, which defaults to
|
||||
"now".
|
||||
|
||||
'''
|
||||
sesh: Interval
|
||||
for sesh in iter_sessions(con_deats):
|
||||
if is_current_time_in_range(
|
||||
sesh=sesh,
|
||||
when=when,
|
||||
):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def is_venue_closure(
|
||||
gap: Interval,
|
||||
con_deats: ContractDetails,
|
||||
time_step_s: int,
|
||||
) -> bool:
|
||||
'''
|
||||
Check if a provided time-`gap` is just an (expected) trading
|
||||
venue closure period.
|
||||
|
||||
'''
|
||||
open: Time
|
||||
close: Time
|
||||
open, close = sesh_times(con_deats)
|
||||
|
||||
# ensure times are in mkt-native timezone
|
||||
tz: str = con_deats.timeZoneId
|
||||
start = gap.start.in_tz(tz)
|
||||
start_t = start.time()
|
||||
end = gap.end.in_tz(tz)
|
||||
end_t = end.time()
|
||||
if (
|
||||
(
|
||||
start_t in (
|
||||
close,
|
||||
close.subtract(seconds=time_step_s)
|
||||
)
|
||||
and
|
||||
end_t in (
|
||||
open,
|
||||
open.add(seconds=time_step_s),
|
||||
)
|
||||
)
|
||||
or
|
||||
has_weekend(gap)
|
||||
or
|
||||
has_holiday(
|
||||
con_deats=con_deats,
|
||||
period=gap,
|
||||
)
|
||||
):
|
||||
return True
|
||||
|
||||
# breakpoint()
|
||||
return False
|
||||
|
||||
|
||||
# TODO, put this into `._util` and call it from here!
|
||||
#
|
||||
# NOTE, this was generated by @guille from a gpt5 prompt
|
||||
# and was originally thot to be needed before learning about
|
||||
# `ib_async.contract.ContractDetails._parseSessions()` and
|
||||
# it's downstream meths..
|
||||
#
|
||||
# This is still likely useful to keep for now to parse the
|
||||
# `.tradingHours: str` value manually if we ever decide
|
||||
# to move off `ib_async` and implement our own `trio`/`anyio`
|
||||
# based version Bp
|
||||
#
|
||||
# >attempt to parse the retarted ib "time stampy thing" they
|
||||
# >do for "venue hours" with this.. written by
|
||||
# >gpt5-"thinking",
|
||||
#
|
||||
|
||||
|
||||
def parse_trading_hours(
|
||||
spec: str,
|
||||
tz: TzInfo|None = None
|
||||
) -> dict[
|
||||
date,
|
||||
tuple[datetime, datetime]
|
||||
]|None:
|
||||
'''
|
||||
Parse venue hours like:
|
||||
'YYYYMMDD:HHMM-YYYYMMDD:HHMM;YYYYMMDD:CLOSED;...'
|
||||
|
||||
Returns `dict[date] = (open_dt, close_dt)` or `None` if
|
||||
closed.
|
||||
|
||||
'''
|
||||
if (
|
||||
not isinstance(spec, str)
|
||||
or
|
||||
not spec
|
||||
):
|
||||
raise ValueError('spec must be a non-empty string')
|
||||
|
||||
out: dict[
|
||||
date,
|
||||
tuple[datetime, datetime]
|
||||
]|None = {}
|
||||
|
||||
for part in (p.strip() for p in spec.split(';') if p.strip()):
|
||||
if part.endswith(':CLOSED'):
|
||||
day_s, _ = part.split(':', 1)
|
||||
d = datetime.strptime(day_s, '%Y%m%d').date()
|
||||
out[d] = None
|
||||
continue
|
||||
|
||||
try:
|
||||
start_s, end_s = part.split('-', 1)
|
||||
start_dt = datetime.strptime(start_s, '%Y%m%d:%H%M')
|
||||
end_dt = datetime.strptime(end_s, '%Y%m%d:%H%M')
|
||||
except ValueError as exc:
|
||||
raise ValueError(f'invalid segment: {part}') from exc
|
||||
|
||||
if tz is not None:
|
||||
start_dt = start_dt.replace(tzinfo=tz)
|
||||
end_dt = end_dt.replace(tzinfo=tz)
|
||||
|
||||
out[start_dt.date()] = (start_dt, end_dt)
|
||||
|
||||
return out
|
||||
|
||||
|
||||
# ORIG desired usage,
|
||||
#
|
||||
# TODO, for non-drunk tomorrow,
|
||||
# - call above fn and check that `output[today] is not None`
|
||||
# trading_hrs: dict = parse_trading_hours(
|
||||
# details.tradingHours
|
||||
# )
|
||||
# liq_hrs: dict = parse_trading_hours(
|
||||
# details.liquidHours
|
||||
# )
|
||||
|
|
@ -27,21 +27,18 @@ from typing import (
|
|||
)
|
||||
import time
|
||||
|
||||
import httpx
|
||||
import pendulum
|
||||
import asks
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
import numpy as np
|
||||
import urllib.parse
|
||||
import hashlib
|
||||
import hmac
|
||||
import base64
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
from piker import config
|
||||
from piker.data import (
|
||||
def_iohlcv_fields,
|
||||
match_from_pairs,
|
||||
)
|
||||
from piker.data import def_iohlcv_fields
|
||||
from piker.accounting._mktinfo import (
|
||||
Asset,
|
||||
digits_to_dec,
|
||||
|
|
@ -61,11 +58,6 @@ log = get_logger('piker.brokers.kraken')
|
|||
|
||||
# <uri>/<version>/
|
||||
_url = 'https://api.kraken.com/0'
|
||||
|
||||
_headers: dict[str, str] = {
|
||||
'User-Agent': 'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
|
||||
}
|
||||
|
||||
# TODO: this is the only backend providing this right?
|
||||
# in which case we should drop it from the defaults and
|
||||
# instead make a custom fields descr in this module!
|
||||
|
|
@ -76,18 +68,12 @@ _symbol_info_translation: dict[str, str] = {
|
|||
|
||||
|
||||
def get_config() -> dict[str, Any]:
|
||||
'''
|
||||
Load our section from `piker/brokers.toml`.
|
||||
|
||||
'''
|
||||
conf, path = config.load(
|
||||
conf_name='brokers',
|
||||
touch_if_dne=True,
|
||||
)
|
||||
if (section := conf.get('kraken')) is None:
|
||||
log.warning(
|
||||
f'No config section found for kraken in {path}'
|
||||
)
|
||||
conf, path = config.load()
|
||||
section = conf.get('kraken')
|
||||
|
||||
if section is None:
|
||||
log.warning(f'No config section found for kraken in {path}')
|
||||
return {}
|
||||
|
||||
return section
|
||||
|
|
@ -120,19 +106,16 @@ class InvalidKey(ValueError):
|
|||
|
||||
class Client:
|
||||
|
||||
# assets and mkt pairs are key-ed by kraken's ReST response
|
||||
# symbol-bs_mktids (we call them "X-keys" like fricking
|
||||
# "XXMRZEUR"). these keys used directly since ledger endpoints
|
||||
# return transaction sets keyed with the same set!
|
||||
# symbol mapping from all names to the altname
|
||||
_altnames: dict[str, str] = {}
|
||||
|
||||
# key-ed by kraken's own bs_mktids (like fricking "XXMRZEUR")
|
||||
# with said keys used directly from EP responses so that ledger
|
||||
# parsing can be easily accomplished from both trade-event-msgs
|
||||
# and offline toml files
|
||||
_Assets: dict[str, Asset] = {}
|
||||
_AssetPairs: dict[str, Pair] = {}
|
||||
|
||||
# offer lookup tables for all .altname and .wsname
|
||||
# to the equivalent .xname so that various symbol-schemas
|
||||
# can be mapped to `Pair`s in the tables above.
|
||||
_altnames: dict[str, str] = {}
|
||||
_wsnames: dict[str, str] = {}
|
||||
|
||||
# key-ed by `Pair.bs_fqme: str`, and thus used for search
|
||||
# allowing for lookup using piker's own FQME symbology sys.
|
||||
_pairs: dict[str, Pair] = {}
|
||||
|
|
@ -141,15 +124,16 @@ class Client:
|
|||
def __init__(
|
||||
self,
|
||||
config: dict[str, str],
|
||||
httpx_client: httpx.AsyncClient,
|
||||
|
||||
name: str = '',
|
||||
api_key: str = '',
|
||||
secret: str = ''
|
||||
) -> None:
|
||||
|
||||
self._sesh: httpx.AsyncClient = httpx_client
|
||||
|
||||
self._sesh = asks.Session(connections=4)
|
||||
self._sesh.base_location = _url
|
||||
self._sesh.headers.update({
|
||||
'User-Agent':
|
||||
'krakenex/2.1.0 (+https://github.com/veox/python3-krakenex)'
|
||||
})
|
||||
self._name = name
|
||||
self._api_key = api_key
|
||||
self._secret = secret
|
||||
|
|
@ -171,9 +155,10 @@ class Client:
|
|||
method: str,
|
||||
data: dict,
|
||||
) -> dict[str, Any]:
|
||||
resp: httpx.Response = await self._sesh.post(
|
||||
url=f'/public/{method}',
|
||||
resp = await self._sesh.post(
|
||||
path=f'/public/{method}',
|
||||
json=data,
|
||||
timeout=float('inf')
|
||||
)
|
||||
return resproc(resp, log)
|
||||
|
||||
|
|
@ -184,18 +169,18 @@ class Client:
|
|||
uri_path: str
|
||||
) -> dict[str, Any]:
|
||||
headers = {
|
||||
'Content-Type': 'application/x-www-form-urlencoded',
|
||||
'API-Key': self._api_key,
|
||||
'API-Sign': get_kraken_signature(
|
||||
uri_path,
|
||||
data,
|
||||
self._secret,
|
||||
),
|
||||
'Content-Type':
|
||||
'application/x-www-form-urlencoded',
|
||||
'API-Key':
|
||||
self._api_key,
|
||||
'API-Sign':
|
||||
get_kraken_signature(uri_path, data, self._secret)
|
||||
}
|
||||
resp: httpx.Response = await self._sesh.post(
|
||||
url=f'/private/{method}',
|
||||
resp = await self._sesh.post(
|
||||
path=f'/private/{method}',
|
||||
data=data,
|
||||
headers=headers,
|
||||
timeout=float('inf')
|
||||
)
|
||||
return resproc(resp, log)
|
||||
|
||||
|
|
@ -224,8 +209,8 @@ class Client:
|
|||
by_bsmktid: dict[str, dict] = resp['result']
|
||||
|
||||
balances: dict = {}
|
||||
for xname, bal in by_bsmktid.items():
|
||||
asset: Asset = self._Assets[xname]
|
||||
for respname, bal in by_bsmktid.items():
|
||||
asset: Asset = self._Assets[respname]
|
||||
|
||||
# TODO: which KEY should we use? it's used to index
|
||||
# the `Account.pps: dict` ..
|
||||
|
|
@ -373,7 +358,8 @@ class Client:
|
|||
# 1658347714, 'status': 'Success'}]}
|
||||
|
||||
if xfers:
|
||||
await tractor.pause()
|
||||
import tractor
|
||||
await tractor.pp()
|
||||
|
||||
trans: dict[str, Transaction] = {}
|
||||
for entry in xfers:
|
||||
|
|
@ -381,6 +367,7 @@ class Client:
|
|||
asset_key: str = entry['asset']
|
||||
asset: Asset = self._Assets[asset_key]
|
||||
asset_key: str = asset.name.lower()
|
||||
# asset_key: str = self._altnames[asset_key].lower()
|
||||
|
||||
# XXX: this is in the asset units (likely) so it isn't
|
||||
# quite the same as a commisions cost necessarily..)
|
||||
|
|
@ -486,32 +473,25 @@ class Client:
|
|||
if err:
|
||||
raise SymbolNotFound(pair_patt)
|
||||
|
||||
# NOTE: we try to key pairs by our custom defined
|
||||
# `.bs_fqme` field since we want to offer search over
|
||||
# this pattern set, callers should fill out lookup
|
||||
# tables for kraken's bs_mktid keys to map to these
|
||||
# keys!
|
||||
# XXX: FURTHER kraken's data eng team decided to offer
|
||||
# 3 frickin market-pair-symbol key sets depending on
|
||||
# which frickin API is being used.
|
||||
# Example for the trading pair 'LTC<EUR'
|
||||
# - the "X-key" from rest eps 'XLTCZEUR'
|
||||
# - the "websocket key" from ws msgs is 'LTC/EUR'
|
||||
# - the "altname key" also delivered in pair info is 'LTCEUR'
|
||||
for xkey, data in resp['result'].items():
|
||||
# NOTE: we key pairs by our custom defined `.bs_fqme`
|
||||
# field since we want to offer search over this key
|
||||
# set, callers should fill out lookup tables for
|
||||
# kraken's bs_mktid keys to map to these keys!
|
||||
for key, data in resp['result'].items():
|
||||
pair = Pair(respname=key, **data)
|
||||
|
||||
# NOTE: always cache in pairs tables for faster lookup
|
||||
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
||||
pair = Pair(xname=xkey, **data)
|
||||
# always cache so we can possibly do faster lookup
|
||||
self._AssetPairs[key] = pair
|
||||
|
||||
# register the above `Pair` structs for all
|
||||
# key-sets/monikers: a set of 4 (frickin) tables
|
||||
# acting as a combined surjection of all possible
|
||||
# (and stupid) kraken names to their `Pair` obj.
|
||||
self._AssetPairs[xkey] = pair
|
||||
self._pairs[pair.bs_fqme] = pair
|
||||
self._altnames[pair.altname] = pair
|
||||
self._wsnames[pair.wsname] = pair
|
||||
bs_fqme: str = pair.bs_fqme
|
||||
|
||||
self._pairs[bs_fqme] = pair
|
||||
|
||||
# register the piker pair under all monikers, a giant flat
|
||||
# surjection of all possible (and stupid) kraken names to
|
||||
# the FMQE style piker key.
|
||||
self._altnames[pair.altname] = bs_fqme
|
||||
self._altnames[pair.wsname] = bs_fqme
|
||||
|
||||
if pair_patt is not None:
|
||||
return next(iter(self._pairs.items()))[1]
|
||||
|
|
@ -526,13 +506,12 @@ class Client:
|
|||
Load all market pair info build and cache it for downstream
|
||||
use.
|
||||
|
||||
Multiple pair info lookup tables (like ``._altnames:
|
||||
dict[str, str]``) are created for looking up the
|
||||
piker-native `Pair`-struct from any input of the three
|
||||
(yes, it's that idiotic..) available symbol/pair-key-sets
|
||||
that kraken frickin offers depending on the API including
|
||||
the .altname, .wsname and the weird ass default set they
|
||||
return in ReST responses .xname..
|
||||
An ``._altnames: dict[str, str]`` is available for looking
|
||||
up the piker-native FQME style `Pair.bs_fqme: str` for any
|
||||
input of the three (yes, it's that idiotic) available
|
||||
key-sets that kraken frickin offers depending on the API
|
||||
including the .altname, .wsname and the weird ass default
|
||||
set they return in rest responses..
|
||||
|
||||
'''
|
||||
if (
|
||||
|
|
@ -560,17 +539,13 @@ class Client:
|
|||
await self.get_mkt_pairs()
|
||||
assert self._pairs, '`Client.get_mkt_pairs()` was never called!?'
|
||||
|
||||
matches: dict[str, Pair] = match_from_pairs(
|
||||
pairs=self._pairs,
|
||||
query=pattern.upper(),
|
||||
matches = fuzzy.extractBests(
|
||||
pattern,
|
||||
self._pairs,
|
||||
score_cutoff=50,
|
||||
)
|
||||
|
||||
# repack in .altname-keyed output table
|
||||
return {
|
||||
pair.altname: pair
|
||||
for pair in matches.values()
|
||||
}
|
||||
# repack in dict form
|
||||
return {item[0].altname: item[0] for item in matches}
|
||||
|
||||
async def bars(
|
||||
self,
|
||||
|
|
@ -653,7 +628,7 @@ class Client:
|
|||
def to_bs_fqme(
|
||||
cls,
|
||||
pair_str: str
|
||||
) -> str:
|
||||
) -> tuple[str, Pair]:
|
||||
'''
|
||||
Normalize symbol names to to a 3x3 pair from the global
|
||||
definition map which we build out from the data retreived from
|
||||
|
|
@ -661,7 +636,7 @@ class Client:
|
|||
|
||||
'''
|
||||
try:
|
||||
return cls._altnames[pair_str.upper()].bs_fqme
|
||||
return cls._altnames[pair_str.upper()]
|
||||
except KeyError as ke:
|
||||
raise SymbolNotFound(f'kraken has no {ke.args[0]}')
|
||||
|
||||
|
|
@ -669,36 +644,24 @@ class Client:
|
|||
@acm
|
||||
async def get_client() -> Client:
|
||||
|
||||
conf: dict[str, Any] = get_config()
|
||||
async with httpx.AsyncClient(
|
||||
base_url=_url,
|
||||
headers=_headers,
|
||||
conf = get_config()
|
||||
if conf:
|
||||
client = Client(
|
||||
conf,
|
||||
|
||||
# TODO: is there a way to numerate this?
|
||||
# https://www.python-httpx.org/advanced/clients/#why-use-a-client
|
||||
# connections=4
|
||||
) as trio_client:
|
||||
if conf:
|
||||
client = Client(
|
||||
conf,
|
||||
httpx_client=trio_client,
|
||||
# TODO: don't break these up and just do internal
|
||||
# conf lookups instead..
|
||||
name=conf['key_descr'],
|
||||
api_key=conf['api_key'],
|
||||
secret=conf['secret']
|
||||
)
|
||||
else:
|
||||
client = Client({})
|
||||
|
||||
# TODO: don't break these up and just do internal
|
||||
# conf lookups instead..
|
||||
name=conf['key_descr'],
|
||||
api_key=conf['api_key'],
|
||||
secret=conf['secret']
|
||||
)
|
||||
else:
|
||||
client = Client(
|
||||
conf={},
|
||||
httpx_client=trio_client,
|
||||
)
|
||||
# at startup, load all symbols, and asset info in
|
||||
# batch requests.
|
||||
async with trio.open_nursery() as nurse:
|
||||
nurse.start_soon(client.get_assets)
|
||||
await client.get_mkt_pairs()
|
||||
|
||||
# at startup, load all symbols, and asset info in
|
||||
# batch requests.
|
||||
async with trio.open_nursery() as nurse:
|
||||
nurse.start_soon(client.get_assets)
|
||||
await client.get_mkt_pairs()
|
||||
|
||||
yield client
|
||||
yield client
|
||||
|
|
|
|||
|
|
@ -62,12 +62,9 @@ from piker.clearing._messages import (
|
|||
from piker.brokers import (
|
||||
open_cached_client,
|
||||
)
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from piker.data import open_symcache
|
||||
from .api import (
|
||||
log,
|
||||
Client,
|
||||
BrokerError,
|
||||
)
|
||||
|
|
@ -81,8 +78,6 @@ from .ledger import (
|
|||
verify_balances,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
MsgUnion = Union[
|
||||
BrokerdCancel,
|
||||
BrokerdError,
|
||||
|
|
@ -180,8 +175,9 @@ async def handle_order_requests(
|
|||
|
||||
case {
|
||||
'account': 'kraken.spot' as account,
|
||||
'action': 'buy'|'sell',
|
||||
}:
|
||||
'action': action,
|
||||
} if action in {'buy', 'sell'}:
|
||||
|
||||
# validate
|
||||
order = BrokerdOrder(**msg)
|
||||
|
||||
|
|
@ -266,12 +262,6 @@ async def handle_order_requests(
|
|||
} | extra
|
||||
|
||||
log.info(f'Submitting WS order request:\n{pformat(req)}')
|
||||
|
||||
# NOTE HOWTO, debug order requests
|
||||
#
|
||||
# if 'XRP' in pair:
|
||||
# await tractor.pause()
|
||||
|
||||
await ws.send_msg(req)
|
||||
|
||||
# placehold for sanity checking in relay loop
|
||||
|
|
@ -417,7 +407,7 @@ def trades2pps(
|
|||
# included?
|
||||
account='kraken.' + acctid,
|
||||
symbol=p.mkt.fqme,
|
||||
size=p.cumsize,
|
||||
size=p.size,
|
||||
avg_price=p.ppu,
|
||||
currency='',
|
||||
)
|
||||
|
|
@ -436,15 +426,9 @@ def trades2pps(
|
|||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
async with (
|
||||
# TODO: maybe bind these together and deliver
|
||||
# a tuple from `.open_cached_client()`?
|
||||
|
|
@ -529,7 +513,6 @@ async def open_trade_dialog(
|
|||
ledger_trans: dict[str, Transaction] = await norm_trade_records(
|
||||
ledger,
|
||||
client,
|
||||
api_name_set='xname',
|
||||
)
|
||||
|
||||
if not acnt.pps:
|
||||
|
|
@ -551,7 +534,6 @@ async def open_trade_dialog(
|
|||
api_trans: dict[str, Transaction] = await norm_trade_records(
|
||||
tids2trades,
|
||||
client,
|
||||
api_name_set='xname',
|
||||
)
|
||||
|
||||
# retrieve kraken reported balances
|
||||
|
|
@ -560,7 +542,7 @@ async def open_trade_dialog(
|
|||
# to be reloaded.
|
||||
balances: dict[str, float] = await client.get_balances()
|
||||
|
||||
await verify_balances(
|
||||
verify_balances(
|
||||
acnt,
|
||||
src_fiat,
|
||||
balances,
|
||||
|
|
@ -628,18 +610,18 @@ async def open_trade_dialog(
|
|||
|
||||
# enter relay loop
|
||||
await handle_order_updates(
|
||||
client=client,
|
||||
ws=ws,
|
||||
ws_stream=stream,
|
||||
ems_stream=ems_stream,
|
||||
apiflows=apiflows,
|
||||
ids=ids,
|
||||
reqids2txids=reqids2txids,
|
||||
acnt=acnt,
|
||||
ledger=ledger,
|
||||
acctid=acctid,
|
||||
acc_name=acc_name,
|
||||
token=token,
|
||||
client,
|
||||
ws,
|
||||
stream,
|
||||
ems_stream,
|
||||
apiflows,
|
||||
ids,
|
||||
reqids2txids,
|
||||
acnt,
|
||||
api_trans,
|
||||
acctid,
|
||||
acc_name,
|
||||
token,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -655,8 +637,7 @@ async def handle_order_updates(
|
|||
|
||||
# transaction records which will be updated
|
||||
# on new trade clearing events (aka order "fills")
|
||||
ledger: TransactionLedger,
|
||||
# ledger_trans: dict[str, Transaction],
|
||||
ledger_trans: dict[str, Transaction],
|
||||
acctid: str,
|
||||
acc_name: str,
|
||||
token: str,
|
||||
|
|
@ -716,8 +697,7 @@ async def handle_order_updates(
|
|||
# if tid not in ledger_trans
|
||||
}
|
||||
for tid, trade in trades.items():
|
||||
# assert tid not in ledger_trans
|
||||
assert tid not in ledger
|
||||
assert tid not in ledger_trans
|
||||
txid = trade['ordertxid']
|
||||
reqid = trade.get('userref')
|
||||
|
||||
|
|
@ -763,19 +743,12 @@ async def handle_order_updates(
|
|||
new_trans = await norm_trade_records(
|
||||
trades,
|
||||
client,
|
||||
api_name_set='wsname',
|
||||
)
|
||||
ppmsgs: list[BrokerdPosition] = trades2pps(
|
||||
acnt=acnt,
|
||||
ledger=ledger,
|
||||
acctid=acctid,
|
||||
new_trans=new_trans,
|
||||
ppmsgs = trades2pps(
|
||||
acnt,
|
||||
acctid,
|
||||
new_trans,
|
||||
)
|
||||
# ppmsgs = trades2pps(
|
||||
# acnt,
|
||||
# acctid,
|
||||
# new_trans,
|
||||
# )
|
||||
for pp_msg in ppmsgs:
|
||||
await ems_stream.send(pp_msg)
|
||||
|
||||
|
|
@ -1101,8 +1074,6 @@ async def handle_order_updates(
|
|||
f'Failed to {action} order {reqid}:\n'
|
||||
f'{errmsg}'
|
||||
)
|
||||
# if tractor._state.debug_mode():
|
||||
# await tractor.pause()
|
||||
|
||||
symbol: str = 'N/A'
|
||||
if chain := apiflows.get(reqid):
|
||||
|
|
|
|||
|
|
@ -64,19 +64,9 @@ def norm_trade(
|
|||
'sell': -1,
|
||||
}[record['type']]
|
||||
|
||||
# NOTE: this value may be either the websocket OR the rest schema
|
||||
# so we need to detect the key format and then choose the
|
||||
# correct symbol lookup table to evetually get a ``Pair``..
|
||||
# See internals of `Client.asset_pairs()` for deats!
|
||||
src_pair_key: str = record['pair']
|
||||
rest_pair_key: str = record['pair']
|
||||
pair: Pair = pairs[rest_pair_key]
|
||||
|
||||
# XXX: kraken's data engineering is soo bad they require THREE
|
||||
# different pair schemas (more or less seemingly tied to
|
||||
# transport-APIs)..LITERALLY they return different market id
|
||||
# pairs in the ledger endpoints vs. the websocket event subs..
|
||||
# lookup pair using appropriately provided tabled depending
|
||||
# on API-key-schema..
|
||||
pair: Pair = pairs[src_pair_key]
|
||||
fqme: str = pair.bs_fqme.lower() + '.kraken'
|
||||
|
||||
return Transaction(
|
||||
|
|
@ -93,7 +83,6 @@ def norm_trade(
|
|||
async def norm_trade_records(
|
||||
ledger: dict[str, Any],
|
||||
client: Client,
|
||||
api_name_set: str = 'xname',
|
||||
|
||||
) -> dict[str, Transaction]:
|
||||
'''
|
||||
|
|
@ -108,16 +97,11 @@ async def norm_trade_records(
|
|||
# mkt: MktPair = (await get_mkt_info(manual_fqme))[0]
|
||||
# fqme: str = mkt.fqme
|
||||
# assert fqme == manual_fqme
|
||||
pairs: dict[str, Pair] = {
|
||||
'xname': client._AssetPairs,
|
||||
'wsname': client._wsnames,
|
||||
'altname': client._altnames,
|
||||
}[api_name_set]
|
||||
|
||||
records[tid] = norm_trade(
|
||||
tid,
|
||||
record,
|
||||
pairs=pairs,
|
||||
pairs=client._AssetPairs,
|
||||
)
|
||||
|
||||
return records
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ Symbology defs and search.
|
|||
from decimal import Decimal
|
||||
|
||||
import tractor
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
|
||||
from piker._cacheables import (
|
||||
async_lifo_cache,
|
||||
|
|
@ -40,14 +41,9 @@ from piker.accounting._mktinfo import (
|
|||
)
|
||||
|
||||
|
||||
# https://www.kraken.com/features/api#get-tradable-pairs
|
||||
class Pair(Struct):
|
||||
'''
|
||||
A tradable asset pair as schema-defined by,
|
||||
|
||||
https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs
|
||||
|
||||
'''
|
||||
xname: str # idiotic bs_mktid equiv i guess?
|
||||
respname: str # idiotic bs_mktid equiv i guess?
|
||||
altname: str # alternate pair name
|
||||
wsname: str # WebSocket pair name (if available)
|
||||
aclass_base: str # asset class of base component
|
||||
|
|
@ -57,6 +53,7 @@ class Pair(Struct):
|
|||
lot: str # volume lot size
|
||||
|
||||
cost_decimals: int
|
||||
costmin: float
|
||||
pair_decimals: int # scaling decimal places for pair
|
||||
lot_decimals: int # scaling decimal places for volume
|
||||
|
||||
|
|
@ -82,7 +79,6 @@ class Pair(Struct):
|
|||
tick_size: float # min price step size
|
||||
status: str
|
||||
|
||||
costmin: str|None = None # XXX, only some mktpairs?
|
||||
short_position_limit: float = 0
|
||||
long_position_limit: float = float('inf')
|
||||
|
||||
|
|
@ -98,7 +94,7 @@ class Pair(Struct):
|
|||
make up their minds on a better key set XD
|
||||
|
||||
'''
|
||||
return self.xname
|
||||
return self.respname
|
||||
|
||||
@property
|
||||
def price_tick(self) -> Decimal:
|
||||
|
|
@ -140,10 +136,19 @@ async def open_symbol_search(ctx: tractor.Context) -> None:
|
|||
await ctx.started(cache)
|
||||
|
||||
async with ctx.open_stream() as stream:
|
||||
|
||||
async for pattern in stream:
|
||||
await stream.send(
|
||||
await client.search_symbols(pattern)
|
||||
|
||||
matches = fuzzy.extractBests(
|
||||
pattern,
|
||||
client._pairs,
|
||||
score_cutoff=50,
|
||||
)
|
||||
# repack in dict form
|
||||
await stream.send({
|
||||
pair[0].altname: pair[0]
|
||||
for pair in matches
|
||||
})
|
||||
|
||||
|
||||
@async_lifo_cache()
|
||||
|
|
|
|||
|
|
@ -16,9 +16,10 @@
|
|||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
Kucoin cex API backend.
|
||||
Kucoin broker backend
|
||||
|
||||
'''
|
||||
|
||||
from contextlib import (
|
||||
asynccontextmanager as acm,
|
||||
aclosing,
|
||||
|
|
@ -40,8 +41,9 @@ from typing import (
|
|||
import wsproto
|
||||
from uuid import uuid4
|
||||
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
from trio_typing import TaskStatus
|
||||
import httpx
|
||||
import asks
|
||||
from bidict import bidict
|
||||
import numpy as np
|
||||
import pendulum
|
||||
|
|
@ -62,11 +64,8 @@ from piker._cacheables import (
|
|||
)
|
||||
from piker.log import get_logger
|
||||
from piker.data.validate import FeedInit
|
||||
from piker.types import Struct # NOTE, this is already a `tractor.msg.Struct`
|
||||
from piker.data import (
|
||||
def_iohlcv_fields,
|
||||
match_from_pairs,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.data import def_iohlcv_fields
|
||||
from piker.data._web_bs import (
|
||||
open_autorecon_ws,
|
||||
NoBsWs,
|
||||
|
|
@ -98,18 +97,9 @@ class KucoinMktPair(Struct, frozen=True):
|
|||
def size_tick(self) -> Decimal:
|
||||
return Decimal(str(self.quoteMinSize))
|
||||
|
||||
callauctionFirstStageStartTime: None|float
|
||||
callauctionIsEnabled: bool
|
||||
callauctionPriceCeiling: float|None
|
||||
callauctionPriceFloor: float|None
|
||||
callauctionSecondStageStartTime: float|None
|
||||
callauctionThirdStageStartTime: float|None
|
||||
|
||||
enableTrading: bool
|
||||
feeCategory: int
|
||||
feeCurrency: str
|
||||
isMarginEnabled: bool
|
||||
makerFeeCoefficient: float
|
||||
market: str
|
||||
minFunds: float
|
||||
name: str
|
||||
|
|
@ -119,10 +109,7 @@ class KucoinMktPair(Struct, frozen=True):
|
|||
quoteIncrement: float
|
||||
quoteMaxSize: float
|
||||
quoteMinSize: float
|
||||
st: bool
|
||||
symbol: str # our bs_mktid, kucoin's internal id
|
||||
takerFeeCoefficient: float
|
||||
tradingStartTime: float|None
|
||||
|
||||
|
||||
class AccountTrade(Struct, frozen=True):
|
||||
|
|
@ -223,12 +210,8 @@ def get_config() -> BrokerConfig | None:
|
|||
|
||||
class Client:
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
httpx_client: httpx.AsyncClient,
|
||||
) -> None:
|
||||
self._http: httpx.AsyncClient = httpx_client
|
||||
self._config: BrokerConfig|None = get_config()
|
||||
def __init__(self) -> None:
|
||||
self._config: BrokerConfig | None = get_config()
|
||||
self._pairs: dict[str, KucoinMktPair] = {}
|
||||
self._fqmes2mktids: bidict[str, str] = bidict()
|
||||
self._bars: list[list[float]] = []
|
||||
|
|
@ -242,24 +225,18 @@ class Client:
|
|||
|
||||
) -> dict[str, str | bytes]:
|
||||
'''
|
||||
Generate authenticated request headers:
|
||||
|
||||
Generate authenticated request headers
|
||||
https://docs.kucoin.com/#authentication
|
||||
https://www.kucoin.com/docs/basic-info/connection-method/authentication/creating-a-request
|
||||
https://www.kucoin.com/docs/basic-info/connection-method/authentication/signing-a-message
|
||||
|
||||
'''
|
||||
|
||||
if not self._config:
|
||||
raise ValueError(
|
||||
'No config found when trying to send authenticated request'
|
||||
)
|
||||
'No config found when trying to send authenticated request')
|
||||
|
||||
str_to_sign = (
|
||||
str(int(time.time() * 1000))
|
||||
+
|
||||
action
|
||||
+
|
||||
f'/api/{api}/{endpoint.lstrip("/")}'
|
||||
+ action + f'/api/{api}/{endpoint.lstrip("/")}'
|
||||
)
|
||||
|
||||
signature = base64.b64encode(
|
||||
|
|
@ -270,7 +247,6 @@ class Client:
|
|||
).digest()
|
||||
)
|
||||
|
||||
# TODO: can we cache this between calls?
|
||||
passphrase = base64.b64encode(
|
||||
hmac.new(
|
||||
self._config.key_secret.encode('utf-8'),
|
||||
|
|
@ -292,10 +268,8 @@ class Client:
|
|||
self,
|
||||
action: Literal['POST', 'GET'],
|
||||
endpoint: str,
|
||||
|
||||
api: str = 'v2',
|
||||
headers: dict = {},
|
||||
|
||||
) -> Any:
|
||||
'''
|
||||
Generic request wrapper for Kucoin API
|
||||
|
|
@ -308,19 +282,14 @@ class Client:
|
|||
api,
|
||||
)
|
||||
|
||||
req_meth: Callable = getattr(
|
||||
self._http,
|
||||
action.lower(),
|
||||
)
|
||||
res = await req_meth(
|
||||
url=f'/{api}/{endpoint}',
|
||||
headers=headers,
|
||||
)
|
||||
json: dict = res.json()
|
||||
if (data := json.get('data')) is not None:
|
||||
return data
|
||||
api_url = f'https://api.kucoin.com/api/{api}/{endpoint}'
|
||||
|
||||
res = await asks.request(action, api_url, headers=headers)
|
||||
|
||||
json = res.json()
|
||||
if 'data' in json:
|
||||
return json['data']
|
||||
else:
|
||||
api_url: str = self._http.base_url
|
||||
log.error(
|
||||
f'Error making request to {api_url} ->\n'
|
||||
f'{pformat(res)}'
|
||||
|
|
@ -340,7 +309,7 @@ class Client:
|
|||
'''
|
||||
token_type = 'private' if private else 'public'
|
||||
try:
|
||||
data: dict[str, Any]|None = await self._request(
|
||||
data: dict[str, Any] | None = await self._request(
|
||||
'POST',
|
||||
endpoint=f'bullet-{token_type}',
|
||||
api='v1'
|
||||
|
|
@ -378,8 +347,8 @@ class Client:
|
|||
currencies: dict[str, Currency] = {}
|
||||
entries: list[dict] = await self._request(
|
||||
'GET',
|
||||
endpoint='currencies',
|
||||
api='v1',
|
||||
endpoint='currencies',
|
||||
)
|
||||
for entry in entries:
|
||||
curr = Currency(**entry).copy()
|
||||
|
|
@ -395,29 +364,20 @@ class Client:
|
|||
dict[str, KucoinMktPair],
|
||||
bidict[str, KucoinMktPair],
|
||||
]:
|
||||
entries = await self._request(
|
||||
'GET',
|
||||
endpoint='symbols',
|
||||
)
|
||||
entries = await self._request('GET', 'symbols')
|
||||
log.info(f' {len(entries)} Kucoin market pairs fetched')
|
||||
|
||||
pairs: dict[str, KucoinMktPair] = {}
|
||||
fqmes2mktids: bidict[str, str] = bidict()
|
||||
for item in entries:
|
||||
try:
|
||||
pair = pairs[item['name']] = KucoinMktPair(**item)
|
||||
except TypeError as te:
|
||||
raise TypeError(
|
||||
'`KucoinMktPair` and reponse fields do not match ??\n'
|
||||
f'{KucoinMktPair.fields_diff(item)}\n'
|
||||
) from te
|
||||
pair = pairs[item['name']] = KucoinMktPair(**item)
|
||||
fqmes2mktids[
|
||||
item['name'].lower().replace('-', '')
|
||||
] = pair.name
|
||||
|
||||
return pairs, fqmes2mktids
|
||||
|
||||
async def get_mkt_pairs(
|
||||
async def cache_pairs(
|
||||
self,
|
||||
update: bool = False,
|
||||
|
||||
|
|
@ -445,27 +405,16 @@ class Client:
|
|||
|
||||
) -> dict[str, KucoinMktPair]:
|
||||
'''
|
||||
Use fuzzy search engine to match against pairs, deliver
|
||||
matching ones.
|
||||
Use fuzzy search to match against all market names.
|
||||
|
||||
'''
|
||||
if not len(self._pairs):
|
||||
await self.get_mkt_pairs()
|
||||
assert self._pairs, '`Client.get_mkt_pairs()` was never called!?'
|
||||
data = await self.cache_pairs()
|
||||
|
||||
matches: dict[str, KucoinMktPair] = match_from_pairs(
|
||||
pairs=self._pairs,
|
||||
# query=pattern.upper(),
|
||||
query=pattern.upper(),
|
||||
score_cutoff=35,
|
||||
limit=limit,
|
||||
matches = fuzzy.extractBests(
|
||||
pattern, data, score_cutoff=35, limit=limit
|
||||
)
|
||||
|
||||
# repack in dict form
|
||||
return {
|
||||
pair.name: pair
|
||||
for pair in matches.values()
|
||||
}
|
||||
return {item[0].name: item[0] for item in matches}
|
||||
|
||||
async def last_trades(self, sym: str) -> list[AccountTrade]:
|
||||
trades = await self._request(
|
||||
|
|
@ -605,21 +554,13 @@ def fqme_to_kucoin_sym(
|
|||
|
||||
@acm
|
||||
async def get_client() -> AsyncGenerator[Client, None]:
|
||||
'''
|
||||
Load an API `Client` preconfigured from user settings
|
||||
client = Client()
|
||||
|
||||
'''
|
||||
async with (
|
||||
httpx.AsyncClient(
|
||||
base_url='https://api.kucoin.com/api',
|
||||
) as trio_client,
|
||||
):
|
||||
client = Client(httpx_client=trio_client)
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(client.get_mkt_pairs)
|
||||
await client.get_currencies()
|
||||
async with trio.open_nursery() as n:
|
||||
n.start_soon(client.cache_pairs)
|
||||
await client.get_currencies()
|
||||
|
||||
yield client
|
||||
yield client
|
||||
|
||||
|
||||
@tractor.context
|
||||
|
|
@ -628,7 +569,7 @@ async def open_symbol_search(
|
|||
) -> None:
|
||||
async with open_cached_client('kucoin') as client:
|
||||
# load all symbols locally for fast search
|
||||
await client.get_mkt_pairs()
|
||||
await client.cache_pairs()
|
||||
await ctx.started()
|
||||
|
||||
async with ctx.open_stream() as stream:
|
||||
|
|
@ -655,7 +596,7 @@ async def open_ping_task(
|
|||
await trio.sleep((ping_interval - 1000) / 1000)
|
||||
await ws.send_msg({'id': connect_id, 'type': 'ping'})
|
||||
|
||||
log.warning('Starting ping task for kucoin ws connection')
|
||||
log.info('Starting ping task for kucoin ws connection')
|
||||
n.start_soon(ping_server)
|
||||
|
||||
yield
|
||||
|
|
@ -667,21 +608,16 @@ async def open_ping_task(
|
|||
async def get_mkt_info(
|
||||
fqme: str,
|
||||
|
||||
) -> tuple[
|
||||
MktPair,
|
||||
KucoinMktPair,
|
||||
]:
|
||||
) -> tuple[MktPair, KucoinMktPair]:
|
||||
'''
|
||||
Query for and return both a `piker.accounting.MktPair` and
|
||||
`KucoinMktPair` from provided `fqme: str`
|
||||
(fully-qualified-market-endpoint).
|
||||
Query for and return a `MktPair` and `KucoinMktPair`.
|
||||
|
||||
'''
|
||||
async with open_cached_client('kucoin') as client:
|
||||
# split off any fqme broker part
|
||||
bs_fqme, _, broker = fqme.partition('.')
|
||||
|
||||
pairs: dict[str, KucoinMktPair] = await client.get_mkt_pairs()
|
||||
pairs: dict[str, KucoinMktPair] = await client.cache_pairs()
|
||||
|
||||
try:
|
||||
# likely search result key which is already in native mkt symbol form
|
||||
|
|
@ -749,8 +685,6 @@ async def stream_quotes(
|
|||
|
||||
log.info(f'Starting up quote stream(s) for {symbols}')
|
||||
for sym_str in symbols:
|
||||
mkt: MktPair
|
||||
pair: KucoinMktPair
|
||||
mkt, pair = await get_mkt_info(sym_str)
|
||||
init_msgs.append(
|
||||
FeedInit(mkt_info=mkt)
|
||||
|
|
@ -758,11 +692,7 @@ async def stream_quotes(
|
|||
|
||||
ws: NoBsWs
|
||||
token, ping_interval = await client._get_ws_token()
|
||||
log.info('API reported ping_interval: {ping_interval}\n')
|
||||
|
||||
connect_id: str = str(uuid4())
|
||||
typ: str
|
||||
quote: dict
|
||||
connect_id = str(uuid4())
|
||||
async with (
|
||||
open_autorecon_ws(
|
||||
(
|
||||
|
|
@ -776,37 +706,20 @@ async def stream_quotes(
|
|||
),
|
||||
) as ws,
|
||||
open_ping_task(ws, ping_interval, connect_id),
|
||||
aclosing(
|
||||
iter_normed_quotes(
|
||||
ws, sym_str
|
||||
)
|
||||
) as iter_quotes,
|
||||
aclosing(stream_messages(ws, sym_str)) as msg_gen,
|
||||
):
|
||||
typ, quote = await anext(iter_quotes)
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
# take care to not unblock here until we get a real
|
||||
# trade quote?
|
||||
# ^TODO, remove this right?
|
||||
# -[ ] what often blocks chart boot/new-feed switching
|
||||
# since we'ere waiting for a live quote instead of just
|
||||
# loading history afap..
|
||||
# |_ XXX, not sure if we require a bit of rework to core
|
||||
# feed init logic or if backends justg gotta be
|
||||
# changed up.. feel like there was some causality
|
||||
# dilema prolly only seen with IB too..
|
||||
# while typ != 'trade':
|
||||
# typ, quote = await anext(iter_quotes)
|
||||
while typ != 'trade':
|
||||
# take care to not unblock here until we get a real
|
||||
# trade quote
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
task_status.started((init_msgs, quote))
|
||||
feed_is_live.set()
|
||||
|
||||
# XXX NOTE, DO NOT include the `.<backend>` suffix!
|
||||
# OW the sampling loop will not broadcast correctly..
|
||||
# since `bus._subscribers.setdefault(bs_fqme, set())`
|
||||
# is used inside `.data.open_feed_bus()` !!!
|
||||
topic: str = mkt.bs_fqme
|
||||
async for typ, quote in iter_quotes:
|
||||
await send_chan.send({topic: quote})
|
||||
async for typ, msg in msg_gen:
|
||||
await send_chan.send({sym_str: msg})
|
||||
|
||||
|
||||
@acm
|
||||
|
|
@ -861,7 +774,7 @@ async def subscribe(
|
|||
)
|
||||
|
||||
|
||||
async def iter_normed_quotes(
|
||||
async def stream_messages(
|
||||
ws: NoBsWs,
|
||||
sym: str,
|
||||
|
||||
|
|
@ -892,9 +805,6 @@ async def iter_normed_quotes(
|
|||
|
||||
yield 'trade', {
|
||||
'symbol': sym,
|
||||
# TODO, is 'last' even used elsewhere/a-good
|
||||
# semantic? can't we just read the ticks with our
|
||||
# .data.ticktools.frame_ticks()`/
|
||||
'last': trade_data.price,
|
||||
'brokerd_ts': last_trade_ts,
|
||||
'ticks': [
|
||||
|
|
@ -987,7 +897,7 @@ async def open_history_client(
|
|||
if end_dt is None:
|
||||
inow = round(time.time())
|
||||
|
||||
log.debug(
|
||||
print(
|
||||
f'difference in time between load and processing'
|
||||
f'{inow - times[-1]}'
|
||||
)
|
||||
|
|
|
|||
|
|
@ -37,12 +37,6 @@ import tractor
|
|||
from async_generator import asynccontextmanager
|
||||
import numpy as np
|
||||
import wrapt
|
||||
|
||||
# TODO, port to `httpx`/`trio-websocket` whenver i get back to
|
||||
# writing a proper ws-api streamer for this backend (since the data
|
||||
# feeds are free now) as per GH feat-req:
|
||||
# https://github.com/pikers/piker/issues/509
|
||||
#
|
||||
import asks
|
||||
|
||||
from ..calc import humanize, percent_change
|
||||
|
|
@ -50,19 +44,13 @@ from . import open_cached_client
|
|||
from piker._cacheables import async_lifo_cache
|
||||
from .. import config
|
||||
from ._util import resproc, BrokerError, SymbolNotFound
|
||||
from piker.log import (
|
||||
from ..log import (
|
||||
colorize_json,
|
||||
)
|
||||
from ._util import (
|
||||
log,
|
||||
get_console_log,
|
||||
)
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
|
||||
_use_practice_account = False
|
||||
_refresh_token_ep = 'https://{}login.questrade.com/oauth2/'
|
||||
|
|
@ -1211,10 +1199,7 @@ async def stream_quotes(
|
|||
# feed_type: str = 'stock',
|
||||
) -> AsyncGenerator[str, Dict[str, Any]]:
|
||||
# XXX: required to propagate ``tractor`` loglevel to piker logging
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(loglevel)
|
||||
|
||||
async with open_cached_client('questrade') as client:
|
||||
if feed_type == 'stock':
|
||||
|
|
|
|||
|
|
@ -30,16 +30,9 @@ import asks
|
|||
from ._util import (
|
||||
resproc,
|
||||
BrokerError,
|
||||
log,
|
||||
)
|
||||
from piker.calc import percent_change
|
||||
from piker.log import (
|
||||
get_logger,
|
||||
)
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
from ..calc import percent_change
|
||||
|
||||
_service_ep = 'https://api.robinhood.com'
|
||||
|
||||
|
|
|
|||
|
|
@ -1,49 +0,0 @@
|
|||
piker.clearing
|
||||
______________
|
||||
trade execution-n-control subsys for both live and paper trading as
|
||||
well as algo-trading manual override/interaction across any backend
|
||||
broker and data provider.
|
||||
|
||||
avail UIs
|
||||
*********
|
||||
|
||||
order ctl
|
||||
---------
|
||||
the `piker.clearing` subsys is exposed mainly though
|
||||
the `piker chart` GUI as a "chart trader" style UX and
|
||||
is automatically enabled whenever a chart is opened.
|
||||
|
||||
.. ^TODO, more prose here!
|
||||
|
||||
the "manual" order control features are exposed via the
|
||||
`piker.ui.order_mode` API and can pretty much always be
|
||||
used (at least) in simulated-trading mode, aka "paper"-mode, and
|
||||
the micro-manual is as follows:
|
||||
|
||||
``order_mode`` (
|
||||
edge triggered activation by any of the following keys,
|
||||
``mouse-click`` on y-level to submit at that price
|
||||
):
|
||||
|
||||
- ``f``/ ``ctl-f`` to stage buy
|
||||
- ``d``/ ``ctl-d`` to stage sell
|
||||
- ``a`` to stage alert
|
||||
|
||||
|
||||
``search_mode`` (
|
||||
``ctl-l`` or ``ctl-space`` to open,
|
||||
``ctl-c`` or ``ctl-space`` to close
|
||||
) :
|
||||
|
||||
- begin typing to have symbol search automatically lookup
|
||||
symbols from all loaded backend (broker) providers
|
||||
- arrow keys and mouse click to navigate selection
|
||||
- vi-like ``ctl-[hjkl]`` for navigation
|
||||
|
||||
|
||||
position (pp) mgmt
|
||||
------------------
|
||||
you can also configure your position allocation limits from the
|
||||
sidepane.
|
||||
|
||||
.. ^TODO, explain and provide tut once more refined!
|
||||
|
|
@ -25,10 +25,7 @@ from typing import TYPE_CHECKING
|
|||
|
||||
import trio
|
||||
import tractor
|
||||
from tractor.trionics import (
|
||||
broadcast_receiver,
|
||||
collapse_eg,
|
||||
)
|
||||
from tractor.trionics import broadcast_receiver
|
||||
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
|
|
@ -171,6 +168,7 @@ class OrderClient(Struct):
|
|||
|
||||
|
||||
async def relay_orders_from_sync_code(
|
||||
|
||||
client: OrderClient,
|
||||
symbol_key: str,
|
||||
to_ems_stream: tractor.MsgStream,
|
||||
|
|
@ -215,7 +213,7 @@ async def relay_orders_from_sync_code(
|
|||
async def open_ems(
|
||||
fqme: str,
|
||||
mode: str = 'live',
|
||||
loglevel: str = 'warning',
|
||||
loglevel: str = 'error',
|
||||
|
||||
) -> tuple[
|
||||
OrderClient, # client
|
||||
|
|
@ -244,11 +242,6 @@ async def open_ems(
|
|||
|
||||
async with maybe_open_emsd(
|
||||
broker,
|
||||
# XXX NOTE, LOL so this determines the daemon `emsd` loglevel
|
||||
# then FYI.. that's kinda wrong no?
|
||||
# -[ ] shouldn't it be set by `pikerd -l` or no?
|
||||
# -[ ] would make a lot more sense to have a subsys ctl for
|
||||
# levels.. like `-l emsd.info` or something?
|
||||
loglevel=loglevel,
|
||||
) as portal:
|
||||
|
||||
|
|
@ -288,11 +281,8 @@ async def open_ems(
|
|||
client._ems_stream = trades_stream
|
||||
|
||||
# start sync code order msg delivery task
|
||||
async with (
|
||||
collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
tn.start_soon(
|
||||
async with trio.open_nursery() as n:
|
||||
n.start_soon(
|
||||
relay_orders_from_sync_code,
|
||||
client,
|
||||
fqme,
|
||||
|
|
@ -308,4 +298,4 @@ async def open_ems(
|
|||
)
|
||||
|
||||
# stop the sync-msg-relay task on exit.
|
||||
tn.cancel_scope.cancel()
|
||||
n.cancel_scope.cancel()
|
||||
|
|
|
|||
|
|
@ -42,12 +42,10 @@ from bidict import bidict
|
|||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from tractor import trionics
|
||||
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
subsys,
|
||||
)
|
||||
from ..accounting._mktinfo import (
|
||||
unpack_fqme,
|
||||
|
|
@ -78,6 +76,7 @@ if TYPE_CHECKING:
|
|||
|
||||
# TODO: numba all of this
|
||||
def mk_check(
|
||||
|
||||
trigger_price: float,
|
||||
known_last: float,
|
||||
action: str,
|
||||
|
|
@ -137,7 +136,7 @@ class DarkBook(Struct):
|
|||
tuple[
|
||||
Callable[[float], bool], # predicate
|
||||
tuple[str, ...], # tickfilter
|
||||
dict|Order, # cmd / msg type
|
||||
dict | Order, # cmd / msg type
|
||||
|
||||
# live submission constraint parameters
|
||||
float, # percent_away max price diff
|
||||
|
|
@ -163,7 +162,7 @@ async def clear_dark_triggers(
|
|||
|
||||
router: Router,
|
||||
brokerd_orders_stream: tractor.MsgStream,
|
||||
quote_stream: tractor.MsgStream,
|
||||
quote_stream: tractor.ReceiveMsgStream, # noqa
|
||||
broker: str,
|
||||
fqme: str,
|
||||
|
||||
|
|
@ -179,7 +178,6 @@ async def clear_dark_triggers(
|
|||
'''
|
||||
# XXX: optimize this for speed!
|
||||
# TODO:
|
||||
# - port to the new ringbuf stuff in `tractor.ipc`!
|
||||
# - numba all this!
|
||||
# - this stream may eventually contain multiple symbols
|
||||
quote_stream._raise_on_lag = False
|
||||
|
|
@ -279,7 +277,7 @@ async def clear_dark_triggers(
|
|||
|
||||
# remove exec-condition from set
|
||||
log.info(f'Removing trigger for {oid}')
|
||||
trigger: tuple|None = execs.pop(oid, None)
|
||||
trigger: tuple | None = execs.pop(oid, None)
|
||||
if not trigger:
|
||||
log.warning(
|
||||
f'trigger for {oid} was already removed!?'
|
||||
|
|
@ -337,8 +335,8 @@ async def open_brokerd_dialog(
|
|||
brokermod: ModuleType,
|
||||
portal: tractor.Portal,
|
||||
exec_mode: str,
|
||||
fqme: str|None = None,
|
||||
loglevel: str|None = None,
|
||||
fqme: str | None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> tuple[
|
||||
tractor.MsgStream,
|
||||
|
|
@ -352,21 +350,9 @@ async def open_brokerd_dialog(
|
|||
broker backend, configuration, or client code usage.
|
||||
|
||||
'''
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name='clearing',
|
||||
)
|
||||
# enable `.accounting` console since normally used by
|
||||
# each `brokerd`.
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name='piker.accounting',
|
||||
)
|
||||
broker: str = brokermod.name
|
||||
|
||||
def mk_paper_ep(
|
||||
loglevel: str,
|
||||
):
|
||||
def mk_paper_ep():
|
||||
from . import _paper_engine as paper_mod
|
||||
|
||||
nonlocal brokermod, exec_mode
|
||||
|
|
@ -401,7 +387,6 @@ async def open_brokerd_dialog(
|
|||
for ep_name in [
|
||||
'open_trade_dialog', # probably final name?
|
||||
'trades_dialogue', # legacy
|
||||
# ^!TODO, rm this since all backends ported no ?!?
|
||||
]:
|
||||
trades_endpoint = getattr(
|
||||
brokermod,
|
||||
|
|
@ -418,21 +403,17 @@ async def open_brokerd_dialog(
|
|||
|
||||
if (
|
||||
trades_endpoint is not None
|
||||
or
|
||||
exec_mode != 'paper'
|
||||
or exec_mode != 'paper'
|
||||
):
|
||||
# open live brokerd trades endpoint
|
||||
open_trades_endpoint = portal.open_context(
|
||||
trades_endpoint,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
|
||||
@acm
|
||||
async def maybe_open_paper_ep():
|
||||
if exec_mode == 'paper':
|
||||
async with mk_paper_ep(
|
||||
loglevel=loglevel,
|
||||
) as msg:
|
||||
async with mk_paper_ep() as msg:
|
||||
yield msg
|
||||
return
|
||||
|
||||
|
|
@ -443,9 +424,7 @@ async def open_brokerd_dialog(
|
|||
# runtime indication that the backend can't support live
|
||||
# order ctrl yet, so boot the paperboi B0
|
||||
if first == 'paper':
|
||||
async with mk_paper_ep(
|
||||
loglevel=loglevel,
|
||||
) as msg:
|
||||
async with mk_paper_ep() as msg:
|
||||
yield msg
|
||||
return
|
||||
else:
|
||||
|
|
@ -521,7 +500,7 @@ class Router(Struct):
|
|||
|
||||
'''
|
||||
# setup at actor spawn time
|
||||
_tn: trio.Nursery
|
||||
nursery: trio.Nursery
|
||||
|
||||
# broker to book map
|
||||
books: dict[str, DarkBook] = {}
|
||||
|
|
@ -674,11 +653,7 @@ class Router(Struct):
|
|||
flume = feed.flumes[fqme]
|
||||
first_quote: dict = flume.first_quote
|
||||
book: DarkBook = self.get_dark_book(broker)
|
||||
|
||||
if not (last := first_quote.get('last')):
|
||||
last: float = flume.rt_shm.array[-1]['close']
|
||||
|
||||
book.lasts[fqme]: float = float(last)
|
||||
book.lasts[fqme]: float = float(first_quote['last'])
|
||||
|
||||
async with self.maybe_open_brokerd_dialog(
|
||||
brokermod=brokermod,
|
||||
|
|
@ -691,7 +666,7 @@ class Router(Struct):
|
|||
# dark book clearing loop, also lives with parent
|
||||
# daemon to allow dark order clearing while no
|
||||
# client is connected.
|
||||
self._tn.start_soon(
|
||||
self.nursery.start_soon(
|
||||
clear_dark_triggers,
|
||||
self,
|
||||
relay.brokerd_stream,
|
||||
|
|
@ -714,7 +689,7 @@ class Router(Struct):
|
|||
|
||||
# spawn a ``brokerd`` order control dialog stream
|
||||
# that syncs lifetime with the parent `emsd` daemon.
|
||||
self._tn.start_soon(
|
||||
self.nursery.start_soon(
|
||||
translate_and_relay_brokerd_events,
|
||||
broker,
|
||||
relay.brokerd_stream,
|
||||
|
|
@ -741,14 +716,13 @@ class Router(Struct):
|
|||
subs = self.subscribers[sub_key]
|
||||
|
||||
sent_some: bool = False
|
||||
for client_stream in subs.copy():
|
||||
for client_stream in subs:
|
||||
try:
|
||||
await client_stream.send(msg)
|
||||
sent_some = True
|
||||
except (
|
||||
trio.ClosedResourceError,
|
||||
trio.BrokenResourceError,
|
||||
tractor.TransportClosed,
|
||||
):
|
||||
to_remove.add(client_stream)
|
||||
log.warning(
|
||||
|
|
@ -780,25 +754,19 @@ _router: Router = None
|
|||
@tractor.context
|
||||
async def _setup_persistent_emsd(
|
||||
ctx: tractor.Context,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> None:
|
||||
|
||||
if loglevel:
|
||||
_log = get_console_log(
|
||||
level=loglevel,
|
||||
name=subsys,
|
||||
)
|
||||
assert _log.name == 'piker.clearing'
|
||||
get_console_log(loglevel)
|
||||
|
||||
global _router
|
||||
|
||||
# open a root "service task-nursery" for the `emsd`-actor
|
||||
async with (
|
||||
trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
_router = Router(_tn=tn)
|
||||
# open a root "service nursery" for the ``emsd`` actor
|
||||
async with trio.open_nursery() as service_nursery:
|
||||
|
||||
_router = Router(nursery=service_nursery)
|
||||
|
||||
# TODO: send back the full set of persistent
|
||||
# orders/execs?
|
||||
|
|
@ -845,7 +813,7 @@ async def translate_and_relay_brokerd_events(
|
|||
f'Rx brokerd trade msg:\n'
|
||||
f'{fmsg}'
|
||||
)
|
||||
status_msg: Status|None = None
|
||||
status_msg: Status | None = None
|
||||
|
||||
match brokerd_msg:
|
||||
# BrokerdPosition
|
||||
|
|
@ -945,17 +913,8 @@ async def translate_and_relay_brokerd_events(
|
|||
}:
|
||||
if (
|
||||
not oid
|
||||
# try to lookup any order dialog by
|
||||
# brokerd-side id..
|
||||
and not (
|
||||
oid := book._ems2brokerd_ids.inverse.get(reqid)
|
||||
)
|
||||
):
|
||||
log.warning(
|
||||
f'Rxed unusable error-msg:\n'
|
||||
f'{brokerd_msg}'
|
||||
)
|
||||
continue
|
||||
oid: str = book._ems2brokerd_ids.inverse[reqid]
|
||||
|
||||
msg = BrokerdError(**brokerd_msg)
|
||||
|
||||
|
|
@ -990,10 +949,7 @@ async def translate_and_relay_brokerd_events(
|
|||
fqme: str = (
|
||||
bdmsg.symbol # might be None
|
||||
or
|
||||
bdmsg.broker_details['flow']
|
||||
# NOTE: what happens in empty case in the
|
||||
# broadcast below? it's a problem?
|
||||
.get('symbol', '')
|
||||
bdmsg.broker_details['flow']['symbol']
|
||||
)
|
||||
|
||||
await router.client_broadcast(
|
||||
|
|
@ -1042,28 +998,14 @@ async def translate_and_relay_brokerd_events(
|
|||
status_msg.brokerd_msg = msg
|
||||
status_msg.src = msg.broker_details['name']
|
||||
|
||||
if not status_msg.req:
|
||||
# likely some order change state?
|
||||
await tractor.pause()
|
||||
else:
|
||||
await router.client_broadcast(
|
||||
status_msg.req.symbol,
|
||||
status_msg,
|
||||
)
|
||||
await router.client_broadcast(
|
||||
status_msg.req.symbol,
|
||||
status_msg,
|
||||
)
|
||||
|
||||
if status == 'closed':
|
||||
log.info(
|
||||
f'Execution is complete!\n'
|
||||
f'oid: {oid!r}\n'
|
||||
)
|
||||
status_msg = book._active.pop(oid, None)
|
||||
if status_msg is None:
|
||||
log.warning(
|
||||
f'Order was already cleared from book ??\n'
|
||||
f'oid: {oid!r}\n'
|
||||
f'\n'
|
||||
f'Maybe the order cancelled before submitted ??\n'
|
||||
)
|
||||
log.info(f'Execution for {oid} is complete!')
|
||||
status_msg = book._active.pop(oid)
|
||||
|
||||
elif status == 'canceled':
|
||||
log.cancel(f'Cancellation for {oid} is complete!')
|
||||
|
|
@ -1228,16 +1170,12 @@ async def process_client_order_cmds(
|
|||
submitting live orders immediately if requested by the client.
|
||||
|
||||
'''
|
||||
# TODO, only allow `msgspec.Struct` form!
|
||||
cmd: dict
|
||||
# cmd: dict
|
||||
async for cmd in client_order_stream:
|
||||
log.info(
|
||||
f'Received order cmd:\n'
|
||||
f'{pformat(cmd)}\n'
|
||||
)
|
||||
log.info(f'Received order cmd:\n{pformat(cmd)}')
|
||||
|
||||
# CAWT DAMN we need struct support!
|
||||
oid: str = str(cmd['oid'])
|
||||
oid = str(cmd['oid'])
|
||||
|
||||
# register this stream as an active order dialog (msg flow) for
|
||||
# this order id such that translated message from the brokerd
|
||||
|
|
@ -1306,7 +1244,7 @@ async def process_client_order_cmds(
|
|||
and status.resp == 'dark_open'
|
||||
):
|
||||
# remove from dark book clearing
|
||||
entry: tuple|None = dark_book.triggers[fqme].pop(oid, None)
|
||||
entry: tuple | None = dark_book.triggers[fqme].pop(oid, None)
|
||||
if entry:
|
||||
(
|
||||
pred,
|
||||
|
|
@ -1343,7 +1281,7 @@ async def process_client_order_cmds(
|
|||
case {
|
||||
'oid': oid,
|
||||
'symbol': fqme,
|
||||
'price': price,
|
||||
'price': trigger_price,
|
||||
'size': size,
|
||||
'action': ('buy' | 'sell') as action,
|
||||
'exec_mode': ('live' | 'paper'),
|
||||
|
|
@ -1375,7 +1313,7 @@ async def process_client_order_cmds(
|
|||
|
||||
symbol=sym,
|
||||
action=action,
|
||||
price=price,
|
||||
price=trigger_price,
|
||||
size=size,
|
||||
account=req.account,
|
||||
)
|
||||
|
|
@ -1397,11 +1335,7 @@ async def process_client_order_cmds(
|
|||
# (``translate_and_relay_brokerd_events()`` above) will
|
||||
# handle relaying the ems side responses back to
|
||||
# the client/cmd sender from this request
|
||||
log.info(
|
||||
f'Sending live order to {broker}:\n'
|
||||
f'{pformat(msg)}'
|
||||
)
|
||||
|
||||
log.info(f'Sending live order to {broker}:\n{pformat(msg)}')
|
||||
await brokerd_order_stream.send(msg)
|
||||
|
||||
# an immediate response should be ``BrokerdOrderAck``
|
||||
|
|
@ -1417,7 +1351,7 @@ async def process_client_order_cmds(
|
|||
case {
|
||||
'oid': oid,
|
||||
'symbol': fqme,
|
||||
'price': price,
|
||||
'price': trigger_price,
|
||||
'size': size,
|
||||
'exec_mode': exec_mode,
|
||||
'action': action,
|
||||
|
|
@ -1445,12 +1379,7 @@ async def process_client_order_cmds(
|
|||
if isnan(last):
|
||||
last = flume.rt_shm.array[-1]['close']
|
||||
|
||||
trigger_price: float = float(price)
|
||||
pred = mk_check(
|
||||
trigger_price,
|
||||
last,
|
||||
action,
|
||||
)
|
||||
pred = mk_check(trigger_price, last, action)
|
||||
|
||||
# NOTE: for dark orders currently we submit
|
||||
# the triggered live order at a price 5 ticks
|
||||
|
|
@ -1557,7 +1486,7 @@ async def maybe_open_trade_relays(
|
|||
loglevel: str = 'info',
|
||||
):
|
||||
|
||||
fqme, relay, feed, client_ready = await _router._tn.start(
|
||||
fqme, relay, feed, client_ready = await _router.nursery.start(
|
||||
_router.open_trade_relays,
|
||||
fqme,
|
||||
exec_mode,
|
||||
|
|
@ -1587,18 +1516,19 @@ async def maybe_open_trade_relays(
|
|||
|
||||
@tractor.context
|
||||
async def _emsd_main(
|
||||
ctx: tractor.Context, # becomes `ems_ctx` below
|
||||
ctx: tractor.Context,
|
||||
fqme: str,
|
||||
exec_mode: str, # ('paper', 'live')
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
) -> tuple[ # `ctx.started()` value!
|
||||
dict[ # positions
|
||||
tuple[str, str], # brokername, acctid
|
||||
) -> tuple[
|
||||
dict[
|
||||
# brokername, acctid
|
||||
tuple[str, str],
|
||||
list[BrokerdPosition],
|
||||
],
|
||||
list[str], # accounts
|
||||
dict[str, Status], # dialogs
|
||||
list[str],
|
||||
dict[str, Status],
|
||||
]:
|
||||
'''
|
||||
EMS (sub)actor entrypoint providing the execution management
|
||||
|
|
@ -1723,5 +1653,5 @@ async def _emsd_main(
|
|||
if not client_streams:
|
||||
log.warning(
|
||||
f'Order dialog is not being monitored:\n'
|
||||
f'{oid!r} <-> {client_stream.chan.aid.reprol()}\n'
|
||||
f'{oid} ->\n{client_stream._ctx.chan.uid}'
|
||||
)
|
||||
|
|
|
|||
|
|
@ -19,7 +19,6 @@ Clearing sub-system message and protocols.
|
|||
|
||||
"""
|
||||
from __future__ import annotations
|
||||
from decimal import Decimal
|
||||
from typing import (
|
||||
Literal,
|
||||
)
|
||||
|
|
@ -72,15 +71,7 @@ class Order(Struct):
|
|||
symbol: str # | MktPair
|
||||
account: str # should we set a default as '' ?
|
||||
|
||||
# https://docs.python.org/3/library/decimal.html#decimal-objects
|
||||
#
|
||||
# ?TODO? decimal usage throughout?
|
||||
# -[ ] possibly leverage the `Encoder(decimal_format='number')`
|
||||
# bit?
|
||||
# |_https://jcristharif.com/msgspec/supported-types.html#decimal
|
||||
# -[ ] should we also use it for .size?
|
||||
#
|
||||
price: Decimal
|
||||
price: float
|
||||
size: float # -ve is "sell", +ve is "buy"
|
||||
|
||||
brokers: list[str] = []
|
||||
|
|
@ -187,7 +178,7 @@ class BrokerdOrder(Struct):
|
|||
time_ns: int
|
||||
|
||||
symbol: str # fqme
|
||||
price: Decimal
|
||||
price: float
|
||||
size: float
|
||||
|
||||
# TODO: if we instead rely on a +ve/-ve size to determine
|
||||
|
|
@ -301,9 +292,6 @@ class BrokerdError(Struct):
|
|||
|
||||
# TODO: yeah, so we REALLY need to completely deprecate
|
||||
# this and use the `.accounting.Position` msg-type instead..
|
||||
# -[ ] an alternative might be to add a `Position.summary() ->
|
||||
# `PositionSummary`-msg that we generate since `Position` has a lot
|
||||
# of fields by default we likely don't want to send over the wire?
|
||||
class BrokerdPosition(Struct):
|
||||
'''
|
||||
Position update event from brokerd.
|
||||
|
|
@ -316,4 +304,3 @@ class BrokerdPosition(Struct):
|
|||
avg_price: float
|
||||
currency: str = ''
|
||||
name: str = 'position'
|
||||
bs_mktid: str|int|None = None
|
||||
|
|
|
|||
|
|
@ -26,7 +26,6 @@ from contextlib import asynccontextmanager as acm
|
|||
from datetime import datetime
|
||||
from operator import itemgetter
|
||||
import itertools
|
||||
from pprint import pformat
|
||||
import time
|
||||
from typing import (
|
||||
Callable,
|
||||
|
|
@ -40,7 +39,6 @@ import trio
|
|||
import tractor
|
||||
|
||||
from piker.brokers import get_brokermod
|
||||
from piker.service import find_service
|
||||
from piker.accounting import (
|
||||
Account,
|
||||
MktPair,
|
||||
|
|
@ -59,9 +57,9 @@ from piker.data import (
|
|||
open_symcache,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.log import (
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ._messages import (
|
||||
BrokerdCancel,
|
||||
|
|
@ -73,8 +71,6 @@ from ._messages import (
|
|||
BrokerdError,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
class PaperBoi(Struct):
|
||||
'''
|
||||
|
|
@ -299,8 +295,6 @@ class PaperBoi(Struct):
|
|||
|
||||
# transmit pp msg to ems
|
||||
pp: Position = self.acnt.pps[bs_mktid]
|
||||
# TODO, this will break if `require_only=True` was passed to
|
||||
# `.update_from_ledger()`
|
||||
|
||||
pp_msg = BrokerdPosition(
|
||||
broker=self.broker,
|
||||
|
|
@ -512,7 +506,7 @@ async def handle_order_requests(
|
|||
reqid = await client.submit_limit(
|
||||
oid=order.oid,
|
||||
symbol=f'{order.symbol}.{client.broker}',
|
||||
price=float(order.price),
|
||||
price=order.price,
|
||||
action=order.action,
|
||||
size=order.size,
|
||||
# XXX: by default 0 tells ``ib_insync`` methods that
|
||||
|
|
@ -552,18 +546,16 @@ _sells: defaultdict[
|
|||
|
||||
@tractor.context
|
||||
async def open_trade_dialog(
|
||||
|
||||
ctx: tractor.Context,
|
||||
broker: str,
|
||||
fqme: str|None = None, # if empty, we only boot broker mode
|
||||
fqme: str | None = None, # if empty, we only boot broker mode
|
||||
loglevel: str = 'warning',
|
||||
|
||||
) -> None:
|
||||
|
||||
# enable piker.clearing console log for *this* `brokerd` subactor
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
# enable piker.clearing console log for *this* subactor
|
||||
get_console_log(loglevel)
|
||||
|
||||
symcache: SymbologyCache
|
||||
async with open_symcache(get_brokermod(broker)) as symcache:
|
||||
|
|
@ -659,7 +651,6 @@ async def open_trade_dialog(
|
|||
# in) use manually constructed table from calling
|
||||
# the `.get_mkt_info()` provider EP above.
|
||||
_mktmap_table=mkt_by_fqme,
|
||||
only_require=list(mkt_by_fqme),
|
||||
)
|
||||
|
||||
pp_msgs: list[BrokerdPosition] = []
|
||||
|
|
@ -705,12 +696,7 @@ async def open_trade_dialog(
|
|||
# sanity check all the mkt infos
|
||||
for fqme, flume in feed.flumes.items():
|
||||
mkt: MktPair = symcache.mktmaps.get(fqme) or mkt_by_fqme[fqme]
|
||||
if mkt != flume.mkt:
|
||||
diff: tuple = mkt - flume.mkt
|
||||
log.warning(
|
||||
'MktPair sig mismatch?\n'
|
||||
f'{pformat(diff)}'
|
||||
)
|
||||
assert mkt == flume.mkt
|
||||
|
||||
get_cost: Callable = getattr(
|
||||
brokermod,
|
||||
|
|
@ -768,7 +754,7 @@ async def open_paperboi(
|
|||
service_name = f'paperboi.{broker}'
|
||||
|
||||
async with (
|
||||
find_service(service_name) as portal,
|
||||
tractor.find_actor(service_name) as portal,
|
||||
tractor.open_nursery() as an,
|
||||
):
|
||||
# NOTE: only spawn if no paperboi already is up since we likely
|
||||
|
|
@ -791,10 +777,8 @@ async def open_paperboi(
|
|||
) as (ctx, first):
|
||||
yield ctx, first
|
||||
|
||||
# ALWAYS tear down connection AND any newly spawned
|
||||
# paperboi actor on exit!
|
||||
# tear down connection and any spawned actor on exit
|
||||
await ctx.cancel()
|
||||
|
||||
if we_spawned:
|
||||
await portal.cancel_actor()
|
||||
|
||||
|
|
|
|||
|
|
@ -28,14 +28,11 @@ from ..log import (
|
|||
from piker.types import Struct
|
||||
subsys: str = 'piker.clearing'
|
||||
|
||||
log = get_logger(
|
||||
name='piker.clearing',
|
||||
)
|
||||
log = get_logger(subsys)
|
||||
|
||||
# TODO, oof doesn't this ignore the `loglevel` then???
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
name='clearing',
|
||||
name=subsys,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,33 +1,30 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) 2018-present Tyler Goodlet
|
||||
# (in stewardship for pikers, everywhere.)
|
||||
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or
|
||||
# modify it under the terms of the GNU Affero General Public
|
||||
# License as published by the Free Software Foundation, either
|
||||
# version 3 of the License, or (at your option) any later version.
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Affero General Public License for more details.
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public
|
||||
# License along with this program. If not, see
|
||||
# <https://www.gnu.org/licenses/>.
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
CLI commons.
|
||||
|
||||
'''
|
||||
import os
|
||||
# from contextlib import AsyncExitStack
|
||||
from contextlib import AsyncExitStack
|
||||
from types import ModuleType
|
||||
|
||||
import click
|
||||
import trio
|
||||
import tractor
|
||||
from tractor._multiaddr import parse_maddr
|
||||
|
||||
from ..log import (
|
||||
get_console_log,
|
||||
|
|
@ -45,95 +42,36 @@ from .. import config
|
|||
log = get_logger('piker.cli')
|
||||
|
||||
|
||||
def load_trans_eps(
|
||||
network: dict | None = None,
|
||||
maddrs: list[tuple] | None = None,
|
||||
|
||||
) -> dict[str, dict[str, dict]]:
|
||||
|
||||
# transport-oriented endpoint multi-addresses
|
||||
eps: dict[
|
||||
str, # service name, eg. `pikerd`, `emsd`..
|
||||
|
||||
# libp2p style multi-addresses parsed into prot layers
|
||||
list[dict[str, str | int]]
|
||||
] = {}
|
||||
|
||||
if (
|
||||
network
|
||||
and
|
||||
not maddrs
|
||||
):
|
||||
# load network section and (attempt to) connect all endpoints
|
||||
# which are reachable B)
|
||||
for key, maddrs in network.items():
|
||||
match key:
|
||||
|
||||
# TODO: resolve table across multiple discov
|
||||
# prots Bo
|
||||
case 'resolv':
|
||||
pass
|
||||
|
||||
case 'pikerd':
|
||||
dname: str = key
|
||||
for maddr in maddrs:
|
||||
layers: dict = parse_maddr(maddr)
|
||||
eps.setdefault(
|
||||
dname,
|
||||
[],
|
||||
).append(layers)
|
||||
|
||||
elif maddrs:
|
||||
# presume user is manually specifying the root actor ep.
|
||||
eps['pikerd'] = [parse_maddr(maddr)]
|
||||
|
||||
return eps
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option('--loglevel', '-l', default='warning', help='Logging level')
|
||||
@click.option('--tl', is_flag=True, help='Enable tractor logging')
|
||||
@click.option('--pdb', is_flag=True, help='Enable tractor debug mode')
|
||||
@click.option('--host', '-h', default=None, help='Host addr to bind')
|
||||
@click.option('--port', '-p', default=None, help='Port number to bind')
|
||||
@click.option(
|
||||
'--loglevel',
|
||||
'-l',
|
||||
default='warning',
|
||||
help='Logging level',
|
||||
)
|
||||
@click.option(
|
||||
'--tl',
|
||||
'--tsdb',
|
||||
is_flag=True,
|
||||
help='Enable tractor-runtime logs',
|
||||
help='Enable local ``marketstore`` instance'
|
||||
)
|
||||
@click.option(
|
||||
'--pdb',
|
||||
'--es',
|
||||
is_flag=True,
|
||||
help='Enable tractor debug mode',
|
||||
)
|
||||
@click.option(
|
||||
'--maddr',
|
||||
'-m',
|
||||
default=None,
|
||||
help='Multiaddrs to bind or contact',
|
||||
help='Enable local ``elasticsearch`` instance'
|
||||
)
|
||||
def pikerd(
|
||||
maddr: list[str] | None,
|
||||
loglevel: str,
|
||||
host: str,
|
||||
port: int,
|
||||
tl: bool,
|
||||
pdb: bool,
|
||||
tsdb: bool,
|
||||
es: bool,
|
||||
):
|
||||
'''
|
||||
Start the "root service actor", `pikerd`, run it until
|
||||
cancellation.
|
||||
|
||||
This "root daemon" operates as the top most service-mngr and
|
||||
subsys-as-subactor supervisor, think of it as the "init proc" of
|
||||
any of any `piker` application or daemon-process tree.
|
||||
Spawn the piker broker-daemon.
|
||||
|
||||
'''
|
||||
# from tractor.devx import maybe_open_crash_handler
|
||||
# with maybe_open_crash_handler(pdb=False):
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
with_tractor_log=tl,
|
||||
)
|
||||
log = get_console_log(loglevel, name='cli')
|
||||
|
||||
if pdb:
|
||||
log.warning((
|
||||
|
|
@ -144,49 +82,46 @@ def pikerd(
|
|||
"\n"
|
||||
))
|
||||
|
||||
# service-actor registry endpoint socket-address set
|
||||
regaddrs: list[tuple[str, int]] = []
|
||||
|
||||
conf, _ = config.load(
|
||||
conf_name='conf',
|
||||
)
|
||||
network: dict = conf.get('network')
|
||||
if (
|
||||
network is None
|
||||
and not maddr
|
||||
):
|
||||
regaddrs = [(
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
)]
|
||||
|
||||
else:
|
||||
eps: dict = load_trans_eps(
|
||||
network,
|
||||
maddr,
|
||||
reg_addr: None | tuple[str, int] = None
|
||||
if host or port:
|
||||
reg_addr = (
|
||||
host or _default_registry_host,
|
||||
int(port) or _default_registry_port,
|
||||
)
|
||||
for layers in eps['pikerd']:
|
||||
regaddrs.append((
|
||||
layers['ipv4']['addr'],
|
||||
layers['tcp']['port'],
|
||||
))
|
||||
|
||||
from .. import service
|
||||
|
||||
async def main():
|
||||
service_mngr: service.Services
|
||||
|
||||
async with (
|
||||
service.open_pikerd(
|
||||
registry_addrs=regaddrs,
|
||||
loglevel=loglevel,
|
||||
debug_mode=pdb,
|
||||
# enable_transports=['uds'],
|
||||
enable_transports=['tcp'],
|
||||
) as service_mngr,
|
||||
registry_addr=reg_addr,
|
||||
|
||||
) as service_mngr, # normally delivers a ``Services`` handle
|
||||
|
||||
AsyncExitStack() as stack,
|
||||
):
|
||||
assert service_mngr
|
||||
# ?TODO? spawn all other sub-actor daemons according to
|
||||
# multiaddress endpoint spec defined by user config
|
||||
if tsdb:
|
||||
dname, conf = await stack.enter_async_context(
|
||||
service.marketstore.start_ahab_daemon(
|
||||
service_mngr,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
)
|
||||
log.info(f'TSDB `{dname}` up with conf:\n{conf}')
|
||||
|
||||
if es:
|
||||
dname, conf = await stack.enter_async_context(
|
||||
service.elastic.start_ahab_daemon(
|
||||
service_mngr,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
)
|
||||
log.info(f'DB `{dname}` up with conf:\n{conf}')
|
||||
|
||||
await trio.sleep_forever()
|
||||
|
||||
trio.run(main)
|
||||
|
|
@ -202,24 +137,8 @@ def pikerd(
|
|||
@click.option('--loglevel', '-l', default='warning', help='Logging level')
|
||||
@click.option('--tl', is_flag=True, help='Enable tractor logging')
|
||||
@click.option('--configdir', '-c', help='Configuration directory')
|
||||
@click.option(
|
||||
'--pdb',
|
||||
is_flag=True,
|
||||
help='Enable runtime debug mode ',
|
||||
)
|
||||
@click.option(
|
||||
'--maddr',
|
||||
'-m',
|
||||
default=None,
|
||||
multiple=True,
|
||||
help='Multiaddr to bind',
|
||||
)
|
||||
@click.option(
|
||||
'--regaddr',
|
||||
'-r',
|
||||
default=None,
|
||||
help='Registrar addr to contact',
|
||||
)
|
||||
@click.option('--host', '-h', default=None, help='Host addr to bind')
|
||||
@click.option('--port', '-p', default=None, help='Port number to bind')
|
||||
@click.pass_context
|
||||
def cli(
|
||||
ctx: click.Context,
|
||||
|
|
@ -227,21 +146,10 @@ def cli(
|
|||
loglevel: str,
|
||||
tl: bool,
|
||||
configdir: str,
|
||||
pdb: bool,
|
||||
|
||||
# TODO: make these list[str] with multiple -m maddr0 -m maddr1
|
||||
maddr: list[str],
|
||||
regaddr: str,
|
||||
host: str,
|
||||
port: int,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
The "root" `piker`-cmd CLI endpoint.
|
||||
|
||||
NOTE, this def generally relies on and requires a sub-cmd to be
|
||||
provided by the user, OW only a `--help` msg (listing said
|
||||
subcmds) will be dumped to console.
|
||||
|
||||
'''
|
||||
if configdir is not None:
|
||||
assert os.path.isdir(configdir), f"`{configdir}` is not a valid path"
|
||||
config._override_config_dir(configdir)
|
||||
|
|
@ -260,20 +168,12 @@ def cli(
|
|||
}
|
||||
assert brokermods
|
||||
|
||||
# TODO: load endpoints from `conf::[network].pikerd`
|
||||
# - pikerd vs. regd, separate registry daemon?
|
||||
# - expose datad vs. brokerd?
|
||||
# - bind emsd with certain perms on public iface?
|
||||
regaddrs: list[tuple[str, int]] = regaddr or [(
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
)]
|
||||
|
||||
# TODO: factor [network] section parsing out from pikerd
|
||||
# above and call it here as well.
|
||||
# if maddr:
|
||||
# for addr in maddr:
|
||||
# layers: dict = parse_maddr(addr)
|
||||
reg_addr: None | tuple[str, int] = None
|
||||
if host or port:
|
||||
reg_addr = (
|
||||
host or _default_registry_host,
|
||||
int(port) or _default_registry_port,
|
||||
)
|
||||
|
||||
ctx.obj.update({
|
||||
'brokers': brokers,
|
||||
|
|
@ -283,12 +183,7 @@ def cli(
|
|||
'log': get_console_log(loglevel),
|
||||
'confdir': config._config_dir,
|
||||
'wl_path': config._watchlists_data_path,
|
||||
'registry_addrs': regaddrs,
|
||||
'pdb': pdb, # debug mode flag
|
||||
|
||||
# TODO: endpoint parsing, pinging and binding
|
||||
# on no existing server.
|
||||
# 'maddrs': maddr,
|
||||
'registry_addr': reg_addr,
|
||||
})
|
||||
|
||||
# allow enabling same loglevel in ``tractor`` machinery
|
||||
|
|
@ -300,93 +195,43 @@ def cli(
|
|||
@click.option('--tl', is_flag=True, help='Enable tractor logging')
|
||||
@click.argument('ports', nargs=-1, required=False)
|
||||
@click.pass_obj
|
||||
def services(
|
||||
config,
|
||||
tl: bool,
|
||||
ports: list[int],
|
||||
):
|
||||
'''
|
||||
List all `piker` "service deamons" to the console in
|
||||
a `json`-table which maps each actor's UID in the form,
|
||||
def services(config, tl, ports):
|
||||
|
||||
`{service_name}.{subservice_name}.{UUID}`
|
||||
|
||||
to its (primary) IPC server address.
|
||||
|
||||
(^TODO, should be its multiaddr form once we support it)
|
||||
|
||||
Note that by convention actors which operate as "headless"
|
||||
processes (those without GUIs/graphics, and which generally
|
||||
parent some noteworthy subsystem) are normally suffixed by
|
||||
a "d" such as,
|
||||
|
||||
- pikerd: the root runtime supervisor
|
||||
- brokerd: a broker-backend order ctl daemon
|
||||
- emsd: the internal dark-clearing and order routing daemon
|
||||
- datad: a data-provider-backend data feed daemon
|
||||
- samplerd: the real-time data sampling and clock-syncing daemon
|
||||
|
||||
"Headed units" are normally just given an obvious app-like name
|
||||
with subactors indexed by `.` such as,
|
||||
- chart: the primary modal charting iface, a Qt app
|
||||
- chart.fsp_0: a financial-sig-proc cascade instance which
|
||||
delivers graphics to a parent `chart` app.
|
||||
- polars_boi: some (presumably) `polars` using console app.
|
||||
|
||||
'''
|
||||
from piker.service import (
|
||||
from ..service import (
|
||||
open_piker_runtime,
|
||||
_default_registry_port,
|
||||
_default_registry_host,
|
||||
)
|
||||
|
||||
# !TODO, mk this to work with UDS!
|
||||
host: str = _default_registry_host
|
||||
host = _default_registry_host
|
||||
if not ports:
|
||||
ports: list[int] = [_default_registry_port]
|
||||
|
||||
addr = tractor._addr.wrap_address(
|
||||
addr=(host, ports[0])
|
||||
)
|
||||
ports = [_default_registry_port]
|
||||
|
||||
async def list_services():
|
||||
nonlocal host
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
name='service_query',
|
||||
loglevel=(
|
||||
config['loglevel']
|
||||
if tl
|
||||
else None
|
||||
),
|
||||
loglevel=config['loglevel'] if tl else None,
|
||||
),
|
||||
tractor.get_registry(
|
||||
addr=addr,
|
||||
tractor.get_arbiter(
|
||||
host=host,
|
||||
port=ports[0]
|
||||
) as portal
|
||||
):
|
||||
registry = await portal.run_from_ns(
|
||||
'self',
|
||||
'get_registry',
|
||||
)
|
||||
registry = await portal.run_from_ns('self', 'get_registry')
|
||||
json_d = {}
|
||||
for key, socket in registry.items():
|
||||
json_d[key] = f'{socket}'
|
||||
|
||||
host, port = socket
|
||||
json_d[key] = f'{host}:{port}'
|
||||
click.echo(f"{colorize_json(json_d)}")
|
||||
|
||||
trio.run(list_services)
|
||||
|
||||
|
||||
def _load_clis() -> None:
|
||||
'''
|
||||
Dynamically load and register all subsys CLI endpoints (at call
|
||||
time).
|
||||
|
||||
NOTE, obviously this is normally expected to be called at
|
||||
`import` time and implicitly relies on our use of various
|
||||
`click`/`typer` decorator APIs.
|
||||
|
||||
'''
|
||||
from ..service import marketstore # noqa
|
||||
from ..service import elastic # noqa
|
||||
from ..brokers import cli # noqa
|
||||
from ..ui import cli # noqa
|
||||
from ..watchlists import cli # noqa
|
||||
|
|
@ -396,5 +241,5 @@ def _load_clis() -> None:
|
|||
from ..accounting import cli # noqa
|
||||
|
||||
|
||||
# load all subsytem cli eps
|
||||
# load downstream cli modules
|
||||
_load_clis()
|
||||
|
|
|
|||
127
piker/config.py
127
piker/config.py
|
|
@ -19,6 +19,7 @@ Platform configuration (files) mgmt.
|
|||
|
||||
"""
|
||||
import platform
|
||||
import sys
|
||||
import os
|
||||
import shutil
|
||||
from typing import (
|
||||
|
|
@ -28,7 +29,6 @@ from typing import (
|
|||
from pathlib import Path
|
||||
|
||||
from bidict import bidict
|
||||
import platformdirs
|
||||
import tomlkit
|
||||
try:
|
||||
import tomllib
|
||||
|
|
@ -41,34 +41,54 @@ from .log import get_logger
|
|||
log = get_logger('broker-config')
|
||||
|
||||
|
||||
# XXX NOTE: orig impl was taken from `click`
|
||||
# |_https://github.com/pallets/click/blob/main/src/click/utils.py#L449
|
||||
#
|
||||
# (since apparently they have some super weirdness with SIGINT and
|
||||
# sudo.. no clue we're probably going to slowly just modify it to our
|
||||
# own version over time..)
|
||||
#
|
||||
# XXX NOTE: taken from ``click`` since apparently they have some
|
||||
# super weirdness with sigint and sudo..no clue
|
||||
# we're probably going to slowly just modify it to our own version over
|
||||
# time..
|
||||
def get_app_dir(
|
||||
app_name: str,
|
||||
roaming: bool = True,
|
||||
force_posix: bool = False,
|
||||
|
||||
) -> str:
|
||||
'''
|
||||
Returns the config folder for the application. The default behavior
|
||||
r"""Returns the config folder for the application. The default behavior
|
||||
is to return whatever is most appropriate for the operating system.
|
||||
|
||||
----
|
||||
NOTE, below is originally from `click` impl fn, we can prolly remove?
|
||||
----
|
||||
To give you an idea, for an app called ``"Foo Bar"``, something like
|
||||
the following folders could be returned:
|
||||
|
||||
Mac OS X:
|
||||
``~/Library/Application Support/Foo Bar``
|
||||
Mac OS X (POSIX):
|
||||
``~/.foo-bar``
|
||||
Unix:
|
||||
``~/.config/foo-bar``
|
||||
Unix (POSIX):
|
||||
``~/.foo-bar``
|
||||
Win XP (roaming):
|
||||
``C:\Documents and Settings\<user>\Local Settings\Application Data\Foo``
|
||||
Win XP (not roaming):
|
||||
``C:\Documents and Settings\<user>\Application Data\Foo Bar``
|
||||
Win 7 (roaming):
|
||||
``C:\Users\<user>\AppData\Roaming\Foo Bar``
|
||||
Win 7 (not roaming):
|
||||
``C:\Users\<user>\AppData\Local\Foo Bar``
|
||||
|
||||
.. versionadded:: 2.0
|
||||
|
||||
:param app_name: the application name. This should be properly capitalized
|
||||
and can contain whitespace.
|
||||
:param roaming: controls if the folder should be roaming or not on Windows.
|
||||
Has no affect otherwise.
|
||||
:param force_posix: if this is set to `True` then on any POSIX system the
|
||||
folder will be stored in the home folder with a leading
|
||||
dot instead of the XDG config home or darwin's
|
||||
application support folder.
|
||||
'''
|
||||
"""
|
||||
|
||||
def _posixify(name):
|
||||
return "-".join(name.split()).lower()
|
||||
|
||||
# NOTE: for testing with `pytest` we leverage the `tmp_dir`
|
||||
# fixture to generate (and clean up) a test-request-specific
|
||||
# directory for isolated configuration files such that,
|
||||
|
|
@ -84,57 +104,44 @@ def get_app_dir(
|
|||
# `tractor`) with the testing dir and check for it whenever we
|
||||
# detect `pytest` is being used (which it isn't under normal
|
||||
# operation).
|
||||
# if "pytest" in sys.modules:
|
||||
# import tractor
|
||||
# actor = tractor.current_actor(err_on_no_runtime=False)
|
||||
# if actor: # runtime is up
|
||||
# rvs = tractor._state._runtime_vars
|
||||
# import pdbp; pdbp.set_trace()
|
||||
# testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
|
||||
# assert testdirpath.exists(), 'piker test harness might be borked!?'
|
||||
# app_name = str(testdirpath)
|
||||
if "pytest" in sys.modules:
|
||||
import tractor
|
||||
actor = tractor.current_actor(err_on_no_runtime=False)
|
||||
if actor: # runtime is up
|
||||
rvs = tractor._state._runtime_vars
|
||||
testdirpath = Path(rvs['piker_vars']['piker_test_dir'])
|
||||
assert testdirpath.exists(), 'piker test harness might be borked!?'
|
||||
app_name = str(testdirpath)
|
||||
|
||||
os_name: str = platform.system()
|
||||
conf_dir: Path = platformdirs.user_config_path()
|
||||
app_dir: Path = conf_dir / app_name
|
||||
|
||||
# ?TODO, from `click`; can remove?
|
||||
if platform.system() == 'Windows':
|
||||
key = "APPDATA" if roaming else "LOCALAPPDATA"
|
||||
folder = os.environ.get(key)
|
||||
if folder is None:
|
||||
folder = os.path.expanduser("~")
|
||||
return os.path.join(folder, app_name)
|
||||
if force_posix:
|
||||
def _posixify(name):
|
||||
return "-".join(name.split()).lower()
|
||||
|
||||
return os.path.join(
|
||||
os.path.expanduser(
|
||||
"~/.{}".format(
|
||||
_posixify(app_name)
|
||||
)
|
||||
)
|
||||
os.path.expanduser("~/.{}".format(_posixify(app_name))))
|
||||
if sys.platform == "darwin":
|
||||
return os.path.join(
|
||||
os.path.expanduser("~/Library/Application Support"), app_name
|
||||
)
|
||||
|
||||
log.info(
|
||||
f'Using user config directory,\n'
|
||||
f'platform.system(): {os_name!r}\n'
|
||||
f'conf_dir: {conf_dir!r}\n'
|
||||
f'app_dir: {conf_dir!r}\n'
|
||||
return os.path.join(
|
||||
os.environ.get("XDG_CONFIG_HOME", os.path.expanduser("~/.config")),
|
||||
_posixify(app_name),
|
||||
)
|
||||
return app_dir
|
||||
|
||||
|
||||
_click_config_dir: Path = Path(get_app_dir('piker'))
|
||||
_config_dir: Path = _click_config_dir
|
||||
_parent_user: str = os.environ.get('SUDO_USER')
|
||||
|
||||
# NOTE: when using `sudo` we attempt to determine the non-root user
|
||||
# and still use their normal config dir.
|
||||
if (
|
||||
(_parent_user := os.environ.get('SUDO_USER'))
|
||||
and
|
||||
_parent_user != 'root'
|
||||
):
|
||||
if _parent_user:
|
||||
non_root_user_dir = Path(
|
||||
os.path.expanduser(f'~{_parent_user}')
|
||||
)
|
||||
root: str = 'root'
|
||||
_ccds: str = str(_click_config_dir) # click config dir as string
|
||||
_ccds: str = str(_click_config_dir) # click config dir string
|
||||
i_tail: int = int(_ccds.rfind(root) + len(root))
|
||||
_config_dir = (
|
||||
non_root_user_dir
|
||||
|
|
@ -234,15 +241,12 @@ def repodir() -> Path:
|
|||
repodir: Path = Path(os.environ.get('GITHUB_WORKSPACE'))
|
||||
confdir: Path = repodir / 'config'
|
||||
|
||||
assert confdir.is_dir(), (
|
||||
f'{confdir} DNE, {repodir} is likely incorrect!'
|
||||
)
|
||||
assert confdir.is_dir(), f'{confdir} DNE, {repodir} is likely incorrect!'
|
||||
return repodir
|
||||
|
||||
|
||||
def load(
|
||||
# NOTE: always appended with .toml suffix
|
||||
conf_name: str = 'conf',
|
||||
conf_name: str = 'brokers', # appended with .toml suffix
|
||||
path: Path | None = None,
|
||||
|
||||
decode: Callable[
|
||||
|
|
@ -250,7 +254,7 @@ def load(
|
|||
MutableMapping,
|
||||
] = tomllib.loads,
|
||||
|
||||
touch_if_dne: bool = True,
|
||||
touch_if_dne: bool = False,
|
||||
|
||||
**tomlkws,
|
||||
|
||||
|
|
@ -259,7 +263,7 @@ def load(
|
|||
Load config file by name.
|
||||
|
||||
If desired config is not in the top level piker-user config path then
|
||||
pass the `path: Path` explicitly.
|
||||
pass the ``path: Path`` explicitly.
|
||||
|
||||
'''
|
||||
# create the $HOME/.config/piker dir if dne
|
||||
|
|
@ -274,8 +278,7 @@ def load(
|
|||
|
||||
if (
|
||||
not path.is_file()
|
||||
and
|
||||
touch_if_dne
|
||||
and touch_if_dne
|
||||
):
|
||||
# only do a template if no path provided,
|
||||
# just touch an empty file with same name.
|
||||
|
|
@ -354,9 +357,7 @@ def load_accounts(
|
|||
|
||||
) -> bidict[str, str | None]:
|
||||
|
||||
conf, path = load(
|
||||
conf_name='brokers',
|
||||
)
|
||||
conf, path = load()
|
||||
accounts = bidict()
|
||||
for provider_name, section in conf.items():
|
||||
accounts_section = section.get('accounts')
|
||||
|
|
|
|||
|
|
@ -43,10 +43,8 @@ from ._symcache import (
|
|||
SymbologyCache,
|
||||
open_symcache,
|
||||
get_symcache,
|
||||
match_from_pairs,
|
||||
)
|
||||
from ._sampling import open_sample_stream
|
||||
from ..types import Struct
|
||||
|
||||
|
||||
__all__: list[str] = [
|
||||
|
|
@ -56,7 +54,6 @@ __all__: list[str] = [
|
|||
'ShmArray',
|
||||
'iterticks',
|
||||
'maybe_open_shm_array',
|
||||
'match_from_pairs',
|
||||
'attach_shm_array',
|
||||
'open_shm_array',
|
||||
'get_shm_token',
|
||||
|
|
@ -65,7 +62,6 @@ __all__: list[str] = [
|
|||
'open_symcache',
|
||||
'open_sample_stream',
|
||||
'get_symcache',
|
||||
'Struct',
|
||||
'SymbologyCache',
|
||||
'types',
|
||||
]
|
||||
|
|
|
|||
|
|
@ -41,11 +41,6 @@ if TYPE_CHECKING:
|
|||
)
|
||||
from piker.toolz import Profiler
|
||||
|
||||
# default gap between bars: "bar gap multiplier"
|
||||
# - 0.5 is no overlap between OC arms,
|
||||
# - 1.0 is full overlap on each neighbor sample
|
||||
BGM: float = 0.16
|
||||
|
||||
|
||||
class IncrementalFormatter(msgspec.Struct):
|
||||
'''
|
||||
|
|
@ -518,7 +513,6 @@ class IncrementalFormatter(msgspec.Struct):
|
|||
|
||||
|
||||
class OHLCBarsFmtr(IncrementalFormatter):
|
||||
|
||||
x_offset: np.ndarray = np.array([
|
||||
-0.5,
|
||||
0,
|
||||
|
|
@ -610,9 +604,8 @@ class OHLCBarsFmtr(IncrementalFormatter):
|
|||
vr: tuple[int, int],
|
||||
|
||||
start: int = 0, # XXX: do we need this?
|
||||
|
||||
# 0.5 is no overlap between arms, 1.0 is full overlap
|
||||
gap: float = BGM,
|
||||
w: float = 0.16,
|
||||
|
||||
) -> tuple[
|
||||
np.ndarray,
|
||||
|
|
@ -629,7 +622,7 @@ class OHLCBarsFmtr(IncrementalFormatter):
|
|||
array[:-1],
|
||||
start,
|
||||
bar_w=self.index_step_size,
|
||||
bar_gap=gap * self.index_step_size,
|
||||
bar_gap=w * self.index_step_size,
|
||||
|
||||
# XXX: don't ask, due to a ``numba`` bug..
|
||||
use_time_index=(self.index_field == 'time'),
|
||||
|
|
|
|||
|
|
@ -33,11 +33,6 @@ from typing import (
|
|||
)
|
||||
|
||||
import tractor
|
||||
from tractor import (
|
||||
Context,
|
||||
MsgStream,
|
||||
Channel,
|
||||
)
|
||||
from tractor.trionics import (
|
||||
maybe_open_nursery,
|
||||
)
|
||||
|
|
@ -58,10 +53,7 @@ if TYPE_CHECKING:
|
|||
from ._sharedmem import (
|
||||
ShmArray,
|
||||
)
|
||||
from .feed import (
|
||||
_FeedsBus,
|
||||
Sub,
|
||||
)
|
||||
from .feed import _FeedsBus
|
||||
|
||||
|
||||
# highest frequency sample step is 1 second by default, though in
|
||||
|
|
@ -80,27 +72,20 @@ class Sampler:
|
|||
This non-instantiated type is meant to be a singleton within
|
||||
a `samplerd` actor-service spawned once by the user wishing to
|
||||
time-step-sample (real-time) quote feeds, see
|
||||
`.service.maybe_open_samplerd()` and the below
|
||||
`register_with_sampler()`.
|
||||
``.service.maybe_open_samplerd()`` and the below
|
||||
``register_with_sampler()``.
|
||||
|
||||
'''
|
||||
service_nursery: None|trio.Nursery = None
|
||||
service_nursery: None | trio.Nursery = None
|
||||
|
||||
# TODO: we could stick these in a composed type to avoid angering
|
||||
# the "i hate module scoped variables crowd" (yawn).
|
||||
# TODO: we could stick these in a composed type to avoid
|
||||
# angering the "i hate module scoped variables crowd" (yawn).
|
||||
ohlcv_shms: dict[float, list[ShmArray]] = {}
|
||||
|
||||
# holds one-task-per-sample-period tasks which are spawned as-needed by
|
||||
# data feed requests with a given detected time step usually from
|
||||
# history loading.
|
||||
incr_task_cs: trio.CancelScope|None = None
|
||||
|
||||
bcast_errors: tuple[Exception] = (
|
||||
trio.BrokenResourceError,
|
||||
trio.ClosedResourceError,
|
||||
trio.EndOfChannel,
|
||||
tractor.TransportClosed,
|
||||
)
|
||||
incr_task_cs: trio.CancelScope | None = None
|
||||
|
||||
# holds all the ``tractor.Context`` remote subscriptions for
|
||||
# a particular sample period increment event: all subscribers are
|
||||
|
|
@ -109,7 +94,7 @@ class Sampler:
|
|||
float,
|
||||
list[
|
||||
float,
|
||||
set[MsgStream]
|
||||
set[tractor.MsgStream]
|
||||
],
|
||||
] = defaultdict(
|
||||
lambda: [
|
||||
|
|
@ -249,8 +234,8 @@ class Sampler:
|
|||
async def broadcast(
|
||||
self,
|
||||
period_s: float,
|
||||
time_stamp: float|None = None,
|
||||
info: dict|None = None,
|
||||
time_stamp: float | None = None,
|
||||
info: dict | None = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -265,17 +250,16 @@ class Sampler:
|
|||
subs: set
|
||||
last_ts, subs = pair
|
||||
|
||||
# NOTE, for debugging pub-sub issues
|
||||
# task = trio.lowlevel.current_task()
|
||||
# log.debug(
|
||||
# f'AlL-SUBS@{period_s!r}: {self.subscribers}\n'
|
||||
# f'PAIR: {pair}\n'
|
||||
# f'TASK: {task}: {id(task)}\n'
|
||||
# f'broadcasting {period_s} -> {last_ts}\n'
|
||||
# f'consumers: {subs}'
|
||||
# )
|
||||
borked: set[MsgStream] = set()
|
||||
sent: set[MsgStream] = set()
|
||||
task = trio.lowlevel.current_task()
|
||||
log.debug(
|
||||
f'SUBS {self.subscribers}\n'
|
||||
f'PAIR {pair}\n'
|
||||
f'TASK: {task}: {id(task)}\n'
|
||||
f'broadcasting {period_s} -> {last_ts}\n'
|
||||
# f'consumers: {subs}'
|
||||
)
|
||||
borked: set[tractor.MsgStream] = set()
|
||||
sent: set[tractor.MsgStream] = set()
|
||||
while True:
|
||||
try:
|
||||
for stream in (subs - sent):
|
||||
|
|
@ -290,12 +274,12 @@ class Sampler:
|
|||
await stream.send(msg)
|
||||
sent.add(stream)
|
||||
|
||||
except self.bcast_errors as err:
|
||||
except (
|
||||
trio.BrokenResourceError,
|
||||
trio.ClosedResourceError
|
||||
):
|
||||
log.error(
|
||||
f'Connection dropped for IPC ctx due to,\n'
|
||||
f'{type(err)!r}\n'
|
||||
f'\n'
|
||||
f'{stream._ctx}'
|
||||
f'{stream._ctx.chan.uid} dropped connection'
|
||||
)
|
||||
borked.add(stream)
|
||||
else:
|
||||
|
|
@ -315,7 +299,7 @@ class Sampler:
|
|||
@classmethod
|
||||
async def broadcast_all(
|
||||
self,
|
||||
info: dict|None = None,
|
||||
info: dict | None = None,
|
||||
) -> None:
|
||||
|
||||
# NOTE: take a copy of subs since removals can happen
|
||||
|
|
@ -330,24 +314,16 @@ class Sampler:
|
|||
|
||||
@tractor.context
|
||||
async def register_with_sampler(
|
||||
ctx: Context,
|
||||
ctx: tractor.Context,
|
||||
period_s: float,
|
||||
shms_by_period: dict[float, dict]|None = None,
|
||||
shms_by_period: dict[float, dict] | None = None,
|
||||
|
||||
open_index_stream: bool = True, # open a 2way stream for sample step msgs?
|
||||
sub_for_broadcasts: bool = True, # sampler side to send step updates?
|
||||
loglevel: str|None = None,
|
||||
|
||||
) -> set[int]:
|
||||
) -> None:
|
||||
|
||||
get_console_log(
|
||||
level=(
|
||||
loglevel
|
||||
or
|
||||
tractor.current_actor().loglevel
|
||||
),
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log(tractor.current_actor().loglevel)
|
||||
incr_was_started: bool = False
|
||||
|
||||
try:
|
||||
|
|
@ -372,12 +348,7 @@ async def register_with_sampler(
|
|||
|
||||
# insert the base 1s period (for OHLC style sampling) into
|
||||
# the increment buffer set to update and shift every second.
|
||||
if (
|
||||
shms_by_period is not None
|
||||
# and
|
||||
# feed_is_live.is_set()
|
||||
# ^TODO? pass it in instead?
|
||||
):
|
||||
if shms_by_period is not None:
|
||||
from ._sharedmem import (
|
||||
attach_shm_array,
|
||||
_Token,
|
||||
|
|
@ -391,17 +362,12 @@ async def register_with_sampler(
|
|||
readonly=False,
|
||||
)
|
||||
shms_by_period[period] = shm
|
||||
Sampler.ohlcv_shms.setdefault(
|
||||
period,
|
||||
[],
|
||||
).append(shm)
|
||||
Sampler.ohlcv_shms.setdefault(period, []).append(shm)
|
||||
|
||||
assert Sampler.ohlcv_shms
|
||||
|
||||
# unblock caller
|
||||
await ctx.started(
|
||||
set(Sampler.ohlcv_shms.keys())
|
||||
)
|
||||
await ctx.started(set(Sampler.ohlcv_shms.keys()))
|
||||
|
||||
if open_index_stream:
|
||||
try:
|
||||
|
|
@ -420,8 +386,7 @@ async def register_with_sampler(
|
|||
finally:
|
||||
if (
|
||||
sub_for_broadcasts
|
||||
and
|
||||
subs
|
||||
and subs
|
||||
):
|
||||
try:
|
||||
subs.remove(stream)
|
||||
|
|
@ -447,7 +412,7 @@ async def register_with_sampler(
|
|||
|
||||
async def spawn_samplerd(
|
||||
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
**extra_tractor_kwargs
|
||||
|
||||
) -> bool:
|
||||
|
|
@ -484,7 +449,6 @@ async def spawn_samplerd(
|
|||
register_with_sampler,
|
||||
period_s=1,
|
||||
sub_for_broadcasts=False,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
return True
|
||||
|
||||
|
|
@ -493,7 +457,8 @@ async def spawn_samplerd(
|
|||
|
||||
@acm
|
||||
async def maybe_open_samplerd(
|
||||
loglevel: str|None = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
**pikerd_kwargs,
|
||||
|
||||
) -> tractor.Portal: # noqa
|
||||
|
|
@ -518,13 +483,13 @@ async def maybe_open_samplerd(
|
|||
@acm
|
||||
async def open_sample_stream(
|
||||
period_s: float,
|
||||
shms_by_period: dict[float, dict]|None = None,
|
||||
shms_by_period: dict[float, dict] | None = None,
|
||||
open_index_stream: bool = True,
|
||||
sub_for_broadcasts: bool = True,
|
||||
loglevel: str|None = None,
|
||||
|
||||
# cache_key: str|None = None,
|
||||
# allow_new_sampler: bool = True,
|
||||
cache_key: str | None = None,
|
||||
allow_new_sampler: bool = True,
|
||||
|
||||
ensure_is_active: bool = False,
|
||||
|
||||
) -> AsyncIterator[dict[str, float]]:
|
||||
|
|
@ -553,15 +518,11 @@ async def open_sample_stream(
|
|||
# yield bistream
|
||||
# else:
|
||||
|
||||
ctx: tractor.Context
|
||||
shm_periods: set[int] # in `int`-seconds
|
||||
async with (
|
||||
# XXX: this should be singleton on a host,
|
||||
# a lone broker-daemon per provider should be
|
||||
# created for all practical purposes
|
||||
maybe_open_samplerd(
|
||||
loglevel=loglevel,
|
||||
) as portal,
|
||||
maybe_open_samplerd() as portal,
|
||||
|
||||
portal.open_context(
|
||||
register_with_sampler,
|
||||
|
|
@ -570,12 +531,11 @@ async def open_sample_stream(
|
|||
'shms_by_period': shms_by_period,
|
||||
'open_index_stream': open_index_stream,
|
||||
'sub_for_broadcasts': sub_for_broadcasts,
|
||||
'loglevel': loglevel,
|
||||
},
|
||||
) as (ctx, shm_periods)
|
||||
) as (ctx, first)
|
||||
):
|
||||
if ensure_is_active:
|
||||
assert len(shm_periods) > 1
|
||||
assert len(first) > 1
|
||||
|
||||
async with (
|
||||
ctx.open_stream(
|
||||
|
|
@ -593,7 +553,8 @@ async def open_sample_stream(
|
|||
|
||||
|
||||
async def sample_and_broadcast(
|
||||
bus: _FeedsBus,
|
||||
|
||||
bus: _FeedsBus, # noqa
|
||||
rt_shm: ShmArray,
|
||||
hist_shm: ShmArray,
|
||||
quote_stream: trio.abc.ReceiveChannel,
|
||||
|
|
@ -613,33 +574,11 @@ async def sample_and_broadcast(
|
|||
|
||||
overruns = Counter()
|
||||
|
||||
# NOTE, only used for debugging live-data-feed issues, though
|
||||
# this should be resolved more correctly in the future using the
|
||||
# new typed-msgspec feats of `tractor`!
|
||||
#
|
||||
# XXX, a multiline nested `dict` formatter (since rn quote-msgs
|
||||
# are just that).
|
||||
# pfmt: Callable[[str], str] = mk_repr()
|
||||
|
||||
# iterate stream delivered by broker
|
||||
async for quotes in quote_stream:
|
||||
# print(quotes)
|
||||
|
||||
# XXX WARNING XXX only enable for debugging bc ow can cost
|
||||
# ALOT of perf with HF-feedz!!!
|
||||
#
|
||||
# log.info(
|
||||
# 'Rx live quotes:\n'
|
||||
# f'{pfmt(quotes)}'
|
||||
# )
|
||||
|
||||
# TODO,
|
||||
# -[ ] `numba` or `cython`-nize this loop possibly?
|
||||
# |_alternatively could we do it in rust somehow by upacking
|
||||
# arrow msgs instead of using `msgspec`?
|
||||
# -[ ] use `msgspec.Struct` support in new typed-msging from
|
||||
# `tractor` to ensure only allowed msgs are transmitted?
|
||||
#
|
||||
# TODO: ``numba`` this!
|
||||
for broker_symbol, quote in quotes.items():
|
||||
# TODO: in theory you can send the IPC msg *before* writing
|
||||
# to the sharedmem array to decrease latency, however, that
|
||||
|
|
@ -710,22 +649,12 @@ async def sample_and_broadcast(
|
|||
# eventually block this producer end of the feed and
|
||||
# thus other consumers still attached.
|
||||
sub_key: str = broker_symbol.lower()
|
||||
subs: set[Sub] = bus.get_subs(sub_key)
|
||||
|
||||
# TODO, figure out how to make this useful whilst
|
||||
# incoporating feed "pausing" ..
|
||||
#
|
||||
# if not subs:
|
||||
# all_bs_fqmes: list[str] = list(
|
||||
# bus._subscribers.keys()
|
||||
# )
|
||||
# log.warning(
|
||||
# f'No subscribers for {brokername!r} live-quote ??\n'
|
||||
# f'broker_symbol: {broker_symbol}\n\n'
|
||||
|
||||
# f'Maybe the backend-sys symbol does not match one of,\n'
|
||||
# f'{pfmt(all_bs_fqmes)}\n'
|
||||
# )
|
||||
subs: list[
|
||||
tuple[
|
||||
tractor.MsgStream | trio.MemorySendChannel,
|
||||
float | None, # tick throttle in Hz
|
||||
]
|
||||
] = bus.get_subs(sub_key)
|
||||
|
||||
# NOTE: by default the broker backend doesn't append
|
||||
# it's own "name" into the fqme schema (but maybe it
|
||||
|
|
@ -734,40 +663,34 @@ async def sample_and_broadcast(
|
|||
fqme: str = f'{broker_symbol}.{brokername}'
|
||||
lags: int = 0
|
||||
|
||||
# XXX TODO XXX: speed up this loop in an AOT compiled
|
||||
# lang (like rust or nim or zig)!
|
||||
# AND/OR instead of doing a fan out to TCP sockets
|
||||
# here, we add a shm-style tick queue which readers can
|
||||
# pull from instead of placing the burden of broadcast
|
||||
# on solely on this `brokerd` actor. see issues:
|
||||
# TODO: speed up this loop in an AOT compiled lang (like
|
||||
# rust or nim or zig) and/or instead of doing a fan out to
|
||||
# TCP sockets here, we add a shm-style tick queue which
|
||||
# readers can pull from instead of placing the burden of
|
||||
# broadcast on solely on this `brokerd` actor. see issues:
|
||||
# - https://github.com/pikers/piker/issues/98
|
||||
# - https://github.com/pikers/piker/issues/107
|
||||
|
||||
# for (stream, tick_throttle) in subs.copy():
|
||||
for sub in subs.copy():
|
||||
ipc: MsgStream = sub.ipc
|
||||
throttle: float = sub.throttle_rate
|
||||
for (stream, tick_throttle) in subs.copy():
|
||||
try:
|
||||
with trio.move_on_after(0.2) as cs:
|
||||
if throttle:
|
||||
send_chan: trio.abc.SendChannel = sub.send_chan
|
||||
|
||||
if tick_throttle:
|
||||
# this is a send mem chan that likely
|
||||
# pushes to the ``uniform_rate_send()`` below.
|
||||
try:
|
||||
send_chan.send_nowait(
|
||||
stream.send_nowait(
|
||||
(fqme, quote)
|
||||
)
|
||||
except trio.WouldBlock:
|
||||
overruns[sub_key] += 1
|
||||
ctx: Context = ipc._ctx
|
||||
chan: Channel = ctx.chan
|
||||
ctx = stream._ctx
|
||||
chan = ctx.chan
|
||||
|
||||
log.warning(
|
||||
f'Feed OVERRUN {sub_key}'
|
||||
f'@{bus.brokername} -> \n'
|
||||
f'feed @ {chan.aid.reprol()}\n'
|
||||
f'throttle = {throttle} Hz'
|
||||
'@{bus.brokername} -> \n'
|
||||
f'feed @ {chan.uid}\n'
|
||||
f'throttle = {tick_throttle} Hz'
|
||||
)
|
||||
|
||||
if overruns[sub_key] > 6:
|
||||
|
|
@ -784,10 +707,10 @@ async def sample_and_broadcast(
|
|||
f'{sub_key}:'
|
||||
f'{ctx.cid}@{chan.uid}'
|
||||
)
|
||||
await ipc.aclose()
|
||||
await stream.aclose()
|
||||
raise trio.BrokenResourceError
|
||||
else:
|
||||
await ipc.send(
|
||||
await stream.send(
|
||||
{fqme: quote}
|
||||
)
|
||||
|
||||
|
|
@ -796,17 +719,21 @@ async def sample_and_broadcast(
|
|||
if lags > 10:
|
||||
await tractor.pause()
|
||||
|
||||
except Sampler.bcast_errors as ipc_err:
|
||||
ctx: Context = ipc._ctx
|
||||
chan: Channel = ctx.chan
|
||||
except (
|
||||
trio.BrokenResourceError,
|
||||
trio.ClosedResourceError,
|
||||
trio.EndOfChannel,
|
||||
):
|
||||
ctx = stream._ctx
|
||||
chan = ctx.chan
|
||||
if ctx:
|
||||
log.warning(
|
||||
f'Dropped `brokerd`-feed for {broker_symbol!r} due to,\n'
|
||||
f'x>) {ctx.cid}@{chan.uid}'
|
||||
f'|_{ipc_err!r}\n\n'
|
||||
'Dropped `brokerd`-quotes-feed connection:\n'
|
||||
f'{broker_symbol}:'
|
||||
f'{ctx.cid}@{chan.uid}'
|
||||
)
|
||||
if sub.throttle_rate:
|
||||
assert ipc._closed
|
||||
if tick_throttle:
|
||||
assert stream._closed
|
||||
|
||||
# XXX: do we need to deregister here
|
||||
# if it's done in the fee bus code?
|
||||
|
|
@ -815,16 +742,17 @@ async def sample_and_broadcast(
|
|||
# since there seems to be some kinda race..
|
||||
bus.remove_subs(
|
||||
sub_key,
|
||||
{sub},
|
||||
{(stream, tick_throttle)},
|
||||
)
|
||||
|
||||
|
||||
async def uniform_rate_send(
|
||||
|
||||
rate: float,
|
||||
quote_stream: trio.abc.ReceiveChannel,
|
||||
stream: MsgStream,
|
||||
stream: tractor.MsgStream,
|
||||
|
||||
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||
task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -842,16 +770,13 @@ async def uniform_rate_send(
|
|||
https://gist.github.com/njsmith/7ea44ec07e901cb78ebe1dd8dd846cb9
|
||||
|
||||
'''
|
||||
# ?TODO? dynamically compute the **actual** approx overhead latency per cycle
|
||||
# instead of this magic # bidinezz?
|
||||
throttle_period: float = 1/rate - 0.000616
|
||||
left_to_sleep: float = throttle_period
|
||||
# TODO: compute the approx overhead latency per cycle
|
||||
left_to_sleep = throttle_period = 1/rate - 0.000616
|
||||
|
||||
# send cycle state
|
||||
first_quote: dict|None
|
||||
first_quote = last_quote = None
|
||||
last_send: float = time.time()
|
||||
diff: float = 0
|
||||
last_send = time.time()
|
||||
diff = 0
|
||||
|
||||
task_status.started()
|
||||
ticks_by_type: dict[
|
||||
|
|
@ -862,28 +787,22 @@ async def uniform_rate_send(
|
|||
clear_types = _tick_groups['clears']
|
||||
|
||||
while True:
|
||||
|
||||
# compute the remaining time to sleep for this throttled cycle
|
||||
left_to_sleep: float = throttle_period - diff
|
||||
left_to_sleep = throttle_period - diff
|
||||
|
||||
if left_to_sleep > 0:
|
||||
cs: trio.CancelScope
|
||||
with trio.move_on_after(left_to_sleep) as cs:
|
||||
sym: str
|
||||
last_quote: dict
|
||||
try:
|
||||
sym, last_quote = await quote_stream.receive()
|
||||
except trio.EndOfChannel:
|
||||
log.exception(
|
||||
f'Live stream for feed for ended?\n'
|
||||
f'<=c\n'
|
||||
f' |_[{stream!r}\n'
|
||||
)
|
||||
log.exception(f"feed for {stream} ended?")
|
||||
break
|
||||
|
||||
diff: float = time.time() - last_send
|
||||
diff = time.time() - last_send
|
||||
|
||||
if not first_quote:
|
||||
first_quote: float = last_quote
|
||||
first_quote = last_quote
|
||||
# first_quote['tbt'] = ticks_by_type
|
||||
|
||||
if (throttle_period - diff) > 0:
|
||||
|
|
@ -944,9 +863,7 @@ async def uniform_rate_send(
|
|||
# TODO: now if only we could sync this to the display
|
||||
# rate timing exactly lul
|
||||
try:
|
||||
await stream.send({
|
||||
sym: first_quote
|
||||
})
|
||||
await stream.send({sym: first_quote})
|
||||
except tractor.RemoteActorError as rme:
|
||||
if rme.type is not tractor._exceptions.StreamOverrun:
|
||||
raise
|
||||
|
|
@ -957,28 +874,19 @@ async def uniform_rate_send(
|
|||
f'{sym}:{ctx.cid}@{chan.uid}'
|
||||
)
|
||||
|
||||
# NOTE: any of these can be raised by `tractor`'s IPC
|
||||
# transport-layer and we want to be highly resilient
|
||||
# to consumers which crash or lose network connection.
|
||||
# I.e. we **DO NOT** want to crash and propagate up to
|
||||
# ``pikerd`` these kinds of errors!
|
||||
except (
|
||||
# NOTE: any of these can be raised by ``tractor``'s IPC
|
||||
# transport-layer and we want to be highly resilient
|
||||
# to consumers which crash or lose network connection.
|
||||
# I.e. we **DO NOT** want to crash and propagate up to
|
||||
# ``pikerd`` these kinds of errors!
|
||||
trio.ClosedResourceError,
|
||||
trio.BrokenResourceError,
|
||||
ConnectionResetError,
|
||||
) + Sampler.bcast_errors as ipc_err:
|
||||
match ipc_err:
|
||||
case trio.EndOfChannel():
|
||||
log.info(
|
||||
f'{stream} terminated by peer,\n'
|
||||
f'{ipc_err!r}'
|
||||
)
|
||||
case _:
|
||||
# if the feed consumer goes down then drop
|
||||
# out of this rate limiter
|
||||
log.warning(
|
||||
f'{stream} closed due to,\n'
|
||||
f'{ipc_err!r}'
|
||||
)
|
||||
|
||||
):
|
||||
# if the feed consumer goes down then drop
|
||||
# out of this rate limiter
|
||||
log.warning(f'{stream} closed')
|
||||
await stream.aclose()
|
||||
return
|
||||
|
||||
|
|
|
|||
|
|
@ -520,12 +520,9 @@ def open_shm_array(
|
|||
|
||||
# "unlink" created shm on process teardown by
|
||||
# pushing teardown calls onto actor context stack
|
||||
stack = tractor.current_actor(
|
||||
err_on_no_runtime=False,
|
||||
).lifetime_stack
|
||||
if stack:
|
||||
stack.callback(shmarr.close)
|
||||
stack.callback(shmarr.destroy)
|
||||
stack = tractor.current_actor().lifetime_stack
|
||||
stack.callback(shmarr.close)
|
||||
stack.callback(shmarr.destroy)
|
||||
|
||||
return shmarr
|
||||
|
||||
|
|
@ -610,10 +607,7 @@ def attach_shm_array(
|
|||
_known_tokens[key] = token
|
||||
|
||||
# "close" attached shm on actor teardown
|
||||
if (actor := tractor.current_actor(
|
||||
err_on_no_runtime=False,
|
||||
)):
|
||||
actor.lifetime_stack.callback(sha.close)
|
||||
tractor.current_actor().lifetime_stack.callback(sha.close)
|
||||
|
||||
return sha
|
||||
|
||||
|
|
|
|||
|
|
@ -31,14 +31,11 @@ from pathlib import Path
|
|||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
Callable,
|
||||
Sequence,
|
||||
Hashable,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
from types import ModuleType
|
||||
|
||||
from rapidfuzz import process as fuzzy
|
||||
from fuzzywuzzy import process as fuzzy
|
||||
import tomli_w # for fast symbol cache writing
|
||||
import tractor
|
||||
import trio
|
||||
|
|
@ -57,7 +54,7 @@ from piker.brokers import (
|
|||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.accounting import (
|
||||
from ..accounting import (
|
||||
Asset,
|
||||
MktPair,
|
||||
)
|
||||
|
|
@ -91,18 +88,6 @@ class SymbologyCache(Struct):
|
|||
# provided by the backend pkg.
|
||||
mktmaps: dict[str, MktPair] = field(default_factory=dict)
|
||||
|
||||
def pformat(self) -> str:
|
||||
return (
|
||||
f'<{type(self).__name__}(\n'
|
||||
f' .mod: {self.mod!r}\n'
|
||||
f' .assets: {len(self.assets)!r}\n'
|
||||
f' .pairs: {len(self.pairs)!r}\n'
|
||||
f' .mktmaps: {len(self.mktmaps)!r}\n'
|
||||
f')>'
|
||||
)
|
||||
|
||||
__repr__ = pformat
|
||||
|
||||
def write_config(self) -> None:
|
||||
|
||||
# put the backend's pair-struct type ref at the top
|
||||
|
|
@ -143,8 +128,8 @@ class SymbologyCache(Struct):
|
|||
- `.get_mkt_pairs()`: returning a table of pair-`Struct`
|
||||
types, custom defined by the particular backend.
|
||||
|
||||
AND, the required `.get_mkt_info()` module-level endpoint
|
||||
which maps `fqme: str` -> `MktPair`s.
|
||||
AND, the required `.get_mkt_info()` module-level endpoint which
|
||||
maps `fqme: str` -> `MktPair`s.
|
||||
|
||||
These tables are then used to fill out the `.assets`, `.pairs` and
|
||||
`.mktmaps` tables on this cache instance, respectively.
|
||||
|
|
@ -162,68 +147,57 @@ class SymbologyCache(Struct):
|
|||
'Implement `Client.get_assets()`!'
|
||||
)
|
||||
|
||||
get_mkt_pairs: Callable|None = getattr(
|
||||
client,
|
||||
'get_mkt_pairs',
|
||||
None,
|
||||
)
|
||||
if not get_mkt_pairs:
|
||||
if get_mkt_pairs := getattr(client, 'get_mkt_pairs', None):
|
||||
|
||||
pairs: dict[str, Struct] = await get_mkt_pairs()
|
||||
for bs_fqme, pair in pairs.items():
|
||||
|
||||
# NOTE: every backend defined pair should
|
||||
# declare it's ns path for roundtrip
|
||||
# serialization lookup.
|
||||
if not getattr(pair, 'ns_path', None):
|
||||
raise TypeError(
|
||||
f'Pair-struct for {self.mod.name} MUST define a '
|
||||
'`.ns_path: str`!\n'
|
||||
f'{pair}'
|
||||
)
|
||||
|
||||
entry = await self.mod.get_mkt_info(pair.bs_fqme)
|
||||
if not entry:
|
||||
continue
|
||||
|
||||
mkt: MktPair
|
||||
pair: Struct
|
||||
mkt, _pair = entry
|
||||
assert _pair is pair, (
|
||||
f'`{self.mod.name}` backend probably has a '
|
||||
'keying-symmetry problem between the pair-`Struct` '
|
||||
'returned from `Client.get_mkt_pairs()`and the '
|
||||
'module level endpoint: `.get_mkt_info()`\n\n'
|
||||
"Here's the struct diff:\n"
|
||||
f'{_pair - pair}'
|
||||
)
|
||||
# NOTE XXX: this means backends MUST implement
|
||||
# a `Struct.bs_mktid: str` field to provide
|
||||
# a native-keyed map to their own symbol
|
||||
# set(s).
|
||||
self.pairs[pair.bs_mktid] = pair
|
||||
|
||||
# NOTE: `MktPair`s are keyed here using piker's
|
||||
# internal FQME schema so that search,
|
||||
# accounting and feed init can be accomplished
|
||||
# a sane, uniform, normalized basis.
|
||||
self.mktmaps[mkt.fqme] = mkt
|
||||
|
||||
self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
|
||||
pair,
|
||||
)
|
||||
|
||||
else:
|
||||
log.warning(
|
||||
'No symbology cache `Pair` support for `{provider}`..\n'
|
||||
'Implement `Client.get_mkt_pairs()`!'
|
||||
)
|
||||
return self
|
||||
|
||||
pairs: dict[str, Struct] = await get_mkt_pairs()
|
||||
if not pairs:
|
||||
log.warning(
|
||||
'No pairs from intial {provider!r} sym-cache request?\n\n'
|
||||
'`Client.get_mkt_pairs()` -> {pairs!r} ?'
|
||||
)
|
||||
return self
|
||||
|
||||
for bs_fqme, pair in pairs.items():
|
||||
if not getattr(pair, 'ns_path', None):
|
||||
# XXX: every backend defined pair must declare
|
||||
# a `.ns_path: tractor.NamespacePath` to enable
|
||||
# roundtrip serialization lookup from a local
|
||||
# cache file.
|
||||
raise TypeError(
|
||||
f'Pair-struct for {self.mod.name} MUST define a '
|
||||
'`.ns_path: str`!\n\n'
|
||||
f'{pair!r}'
|
||||
)
|
||||
|
||||
entry = await self.mod.get_mkt_info(pair.bs_fqme)
|
||||
if not entry:
|
||||
continue
|
||||
|
||||
mkt: MktPair
|
||||
pair: Struct
|
||||
mkt, _pair = entry
|
||||
assert _pair is pair, (
|
||||
f'`{self.mod.name}` backend probably has a '
|
||||
'keying-symmetry problem between the pair-`Struct` '
|
||||
'returned from `Client.get_mkt_pairs()`and the '
|
||||
'module level endpoint: `.get_mkt_info()`\n\n'
|
||||
"Here's the struct diff:\n"
|
||||
f'{_pair - pair}'
|
||||
)
|
||||
# NOTE XXX: this means backends MUST implement
|
||||
# a `Struct.bs_mktid: str` field to provide
|
||||
# a native-keyed map to their own symbol
|
||||
# set(s).
|
||||
self.pairs[pair.bs_mktid] = pair
|
||||
|
||||
# NOTE: `MktPair`s are keyed here using piker's
|
||||
# internal FQME schema so that search,
|
||||
# accounting and feed init can be accomplished
|
||||
# a sane, uniform, normalized basis.
|
||||
self.mktmaps[mkt.fqme] = mkt
|
||||
|
||||
self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
|
||||
pair,
|
||||
)
|
||||
|
||||
return self
|
||||
|
||||
|
|
@ -334,7 +308,7 @@ class SymbologyCache(Struct):
|
|||
matches in a `dict` including the `MktPair` values.
|
||||
|
||||
'''
|
||||
matches = fuzzy.extract(
|
||||
matches = fuzzy.extractBests(
|
||||
pattern,
|
||||
getattr(self, table),
|
||||
score_cutoff=50,
|
||||
|
|
@ -492,43 +466,3 @@ def get_symcache(
|
|||
pdbp.xpm()
|
||||
|
||||
return symcache
|
||||
|
||||
|
||||
def match_from_pairs(
|
||||
pairs: dict[str, Struct],
|
||||
query: str,
|
||||
score_cutoff: int = 50,
|
||||
**extract_kwargs,
|
||||
|
||||
) -> dict[str, Struct]:
|
||||
'''
|
||||
Fuzzy search over a "pairs table" maintained by most backends
|
||||
as part of their symbology-info caching internals.
|
||||
|
||||
Scan the native symbol key set and return best ranked
|
||||
matches back in a new `dict`.
|
||||
|
||||
'''
|
||||
|
||||
# TODO: somehow cache this list (per call) like we were in
|
||||
# `open_symbol_search()`?
|
||||
keys: list[str] = list(pairs)
|
||||
matches: list[tuple[
|
||||
Sequence[Hashable], # matching input key
|
||||
Any, # scores
|
||||
Any,
|
||||
]] = fuzzy.extract(
|
||||
# NOTE: most backends provide keys uppercased
|
||||
query=query,
|
||||
choices=keys,
|
||||
score_cutoff=score_cutoff,
|
||||
**extract_kwargs,
|
||||
)
|
||||
|
||||
# pop and repack pairs in output dict
|
||||
matched_pairs: dict[str, Struct] = {}
|
||||
for item in matches:
|
||||
pair_key: str = item[0]
|
||||
matched_pairs[pair_key] = pairs[pair_key]
|
||||
|
||||
return matched_pairs
|
||||
|
|
|
|||
|
|
@ -0,0 +1,336 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
Financial time series processing utilities usually
|
||||
pertaining to OHLCV style sampled data.
|
||||
|
||||
Routines are generally implemented in either ``numpy`` or
|
||||
``polars`` B)
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
from typing import Literal
|
||||
from math import (
|
||||
ceil,
|
||||
floor,
|
||||
)
|
||||
|
||||
import numpy as np
|
||||
import polars as pl
|
||||
|
||||
from ._sharedmem import ShmArray
|
||||
from ..toolz.profile import (
|
||||
Profiler,
|
||||
pg_profile_enabled,
|
||||
ms_slower_then,
|
||||
)
|
||||
|
||||
|
||||
def slice_from_time(
|
||||
arr: np.ndarray,
|
||||
start_t: float,
|
||||
stop_t: float,
|
||||
step: float, # sampler period step-diff
|
||||
|
||||
) -> slice:
|
||||
'''
|
||||
Calculate array indices mapped from a time range and return them in
|
||||
a slice.
|
||||
|
||||
Given an input array with an epoch `'time'` series entry, calculate
|
||||
the indices which span the time range and return in a slice. Presume
|
||||
each `'time'` step increment is uniform and when the time stamp
|
||||
series contains gaps (the uniform presumption is untrue) use
|
||||
``np.searchsorted()`` binary search to look up the appropriate
|
||||
index.
|
||||
|
||||
'''
|
||||
profiler = Profiler(
|
||||
msg='slice_from_time()',
|
||||
disabled=not pg_profile_enabled(),
|
||||
ms_threshold=ms_slower_then,
|
||||
)
|
||||
|
||||
times = arr['time']
|
||||
t_first = floor(times[0])
|
||||
t_last = ceil(times[-1])
|
||||
|
||||
# the greatest index we can return which slices to the
|
||||
# end of the input array.
|
||||
read_i_max = arr.shape[0]
|
||||
|
||||
# compute (presumed) uniform-time-step index offsets
|
||||
i_start_t = floor(start_t)
|
||||
read_i_start = floor(((i_start_t - t_first) // step)) - 1
|
||||
|
||||
i_stop_t = ceil(stop_t)
|
||||
|
||||
# XXX: edge case -> always set stop index to last in array whenever
|
||||
# the input stop time is detected to be greater then the equiv time
|
||||
# stamp at that last entry.
|
||||
if i_stop_t >= t_last:
|
||||
read_i_stop = read_i_max
|
||||
else:
|
||||
read_i_stop = ceil((i_stop_t - t_first) // step) + 1
|
||||
|
||||
# always clip outputs to array support
|
||||
# for read start:
|
||||
# - never allow a start < the 0 index
|
||||
# - never allow an end index > the read array len
|
||||
read_i_start = min(
|
||||
max(0, read_i_start),
|
||||
read_i_max - 1,
|
||||
)
|
||||
read_i_stop = max(
|
||||
0,
|
||||
min(read_i_stop, read_i_max),
|
||||
)
|
||||
|
||||
# check for larger-then-latest calculated index for given start
|
||||
# time, in which case we do a binary search for the correct index.
|
||||
# NOTE: this is usually the result of a time series with time gaps
|
||||
# where it is expected that each index step maps to a uniform step
|
||||
# in the time stamp series.
|
||||
t_iv_start = times[read_i_start]
|
||||
if (
|
||||
t_iv_start > i_start_t
|
||||
):
|
||||
# do a binary search for the best index mapping to ``start_t``
|
||||
# given we measured an overshoot using the uniform-time-step
|
||||
# calculation from above.
|
||||
|
||||
# TODO: once we start caching these per source-array,
|
||||
# we can just overwrite ``read_i_start`` directly.
|
||||
new_read_i_start = np.searchsorted(
|
||||
times,
|
||||
i_start_t,
|
||||
side='left',
|
||||
)
|
||||
|
||||
# TODO: minimize binary search work as much as possible:
|
||||
# - cache these remap values which compensate for gaps in the
|
||||
# uniform time step basis where we calc a later start
|
||||
# index for the given input ``start_t``.
|
||||
# - can we shorten the input search sequence by heuristic?
|
||||
# up_to_arith_start = index[:read_i_start]
|
||||
|
||||
if (
|
||||
new_read_i_start <= read_i_start
|
||||
):
|
||||
# t_diff = t_iv_start - start_t
|
||||
# print(
|
||||
# f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
|
||||
# f'start_t:{start_t} -> 0index start_t:{t_iv_start}\n'
|
||||
# f'diff: {t_diff}\n'
|
||||
# f'REMAPPED START i: {read_i_start} -> {new_read_i_start}\n'
|
||||
# )
|
||||
read_i_start = new_read_i_start
|
||||
|
||||
t_iv_stop = times[read_i_stop - 1]
|
||||
if (
|
||||
t_iv_stop > i_stop_t
|
||||
):
|
||||
# t_diff = stop_t - t_iv_stop
|
||||
# print(
|
||||
# f"WE'RE CUTTING OUT TIME - STEP:{step}\n"
|
||||
# f'calced iv stop:{t_iv_stop} -> stop_t:{stop_t}\n'
|
||||
# f'diff: {t_diff}\n'
|
||||
# # f'SHOULD REMAP STOP: {read_i_start} -> {new_read_i_start}\n'
|
||||
# )
|
||||
new_read_i_stop = np.searchsorted(
|
||||
times[read_i_start:],
|
||||
# times,
|
||||
i_stop_t,
|
||||
side='right',
|
||||
)
|
||||
|
||||
if (
|
||||
new_read_i_stop <= read_i_stop
|
||||
):
|
||||
read_i_stop = read_i_start + new_read_i_stop + 1
|
||||
|
||||
# sanity checks for range size
|
||||
# samples = (i_stop_t - i_start_t) // step
|
||||
# index_diff = read_i_stop - read_i_start + 1
|
||||
# if index_diff > (samples + 3):
|
||||
# breakpoint()
|
||||
|
||||
# read-relative indexes: gives a slice where `shm.array[read_slc]`
|
||||
# will be the data spanning the input time range `start_t` ->
|
||||
# `stop_t`
|
||||
read_slc = slice(
|
||||
int(read_i_start),
|
||||
int(read_i_stop),
|
||||
)
|
||||
|
||||
profiler(
|
||||
'slicing complete'
|
||||
# f'{start_t} -> {abs_slc.start} | {read_slc.start}\n'
|
||||
# f'{stop_t} -> {abs_slc.stop} | {read_slc.stop}\n'
|
||||
)
|
||||
|
||||
# NOTE: if caller needs absolute buffer indices they can
|
||||
# slice the buffer abs index like so:
|
||||
# index = arr['index']
|
||||
# abs_indx = index[read_slc]
|
||||
# abs_slc = slice(
|
||||
# int(abs_indx[0]),
|
||||
# int(abs_indx[-1]),
|
||||
# )
|
||||
|
||||
return read_slc
|
||||
|
||||
|
||||
def detect_null_time_gap(
|
||||
shm: ShmArray,
|
||||
imargin: int = 1,
|
||||
|
||||
) -> tuple[float, float] | None:
|
||||
'''
|
||||
Detect if there are any zero-epoch stamped rows in
|
||||
the presumed 'time' field-column.
|
||||
|
||||
Filter to the gap and return a surrounding index range.
|
||||
|
||||
NOTE: for now presumes only ONE gap XD
|
||||
|
||||
'''
|
||||
# ensure we read buffer state only once so that ShmArray rt
|
||||
# circular-buffer updates don't cause a indexing/size mismatch.
|
||||
array: np.ndarray = shm.array
|
||||
|
||||
zero_pred: np.ndarray = array['time'] == 0
|
||||
zero_t: np.ndarray = array[zero_pred]
|
||||
|
||||
if zero_t.size:
|
||||
istart, iend = zero_t['index'][[0, -1]]
|
||||
start, end = shm._array['time'][
|
||||
[istart - imargin, iend + imargin]
|
||||
]
|
||||
return (
|
||||
istart - imargin,
|
||||
start,
|
||||
end,
|
||||
iend + imargin,
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
t_unit: Literal = Literal[
|
||||
'days',
|
||||
'hours',
|
||||
'minutes',
|
||||
'seconds',
|
||||
'miliseconds',
|
||||
'microseconds',
|
||||
'nanoseconds',
|
||||
]
|
||||
|
||||
|
||||
def with_dts(
|
||||
df: pl.DataFrame,
|
||||
time_col: str = 'time',
|
||||
) -> pl.DataFrame:
|
||||
'''
|
||||
Insert datetime (casted) columns to a (presumably) OHLC sampled
|
||||
time series with an epoch-time column keyed by ``time_col``.
|
||||
|
||||
'''
|
||||
return df.with_columns([
|
||||
pl.col(time_col).shift(1).suffix('_prev'),
|
||||
pl.col(time_col).diff().alias('s_diff'),
|
||||
pl.from_epoch(pl.col(time_col)).alias('dt'),
|
||||
]).with_columns([
|
||||
pl.from_epoch(pl.col(f'{time_col}_prev')).alias('dt_prev'),
|
||||
pl.col('dt').diff().alias('dt_diff'),
|
||||
]) #.with_columns(
|
||||
# pl.col('dt').diff().dt.days().alias('days_dt_diff'),
|
||||
# )
|
||||
|
||||
|
||||
def detect_time_gaps(
|
||||
df: pl.DataFrame,
|
||||
|
||||
time_col: str = 'time',
|
||||
# epoch sampling step diff
|
||||
expect_period: float = 60,
|
||||
|
||||
# datetime diff unit and gap value
|
||||
# crypto mkts
|
||||
# gap_dt_unit: t_unit = 'minutes',
|
||||
# gap_thresh: int = 1,
|
||||
|
||||
# NOTE: legacy stock mkts have venue operating hours
|
||||
# and thus gaps normally no more then 1-2 days at
|
||||
# a time.
|
||||
# XXX -> must be valid ``polars.Expr.dt.<name>``
|
||||
# TODO: allow passing in a frame of operating hours
|
||||
# durations/ranges for faster legit gap checks.
|
||||
gap_dt_unit: t_unit = 'days',
|
||||
gap_thresh: int = 1,
|
||||
|
||||
) -> pl.DataFrame:
|
||||
'''
|
||||
Filter to OHLC datums which contain sample step gaps.
|
||||
|
||||
For eg. legacy markets which have venue close gaps and/or
|
||||
actual missing data segments.
|
||||
|
||||
'''
|
||||
return (
|
||||
with_dts(df)
|
||||
.filter(
|
||||
pl.col('s_diff').abs() > expect_period
|
||||
)
|
||||
.filter(
|
||||
getattr(
|
||||
pl.col('dt_diff').dt,
|
||||
gap_dt_unit,
|
||||
)().abs() > gap_thresh
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def detect_price_gaps(
|
||||
df: pl.DataFrame,
|
||||
gt_multiplier: float = 2.,
|
||||
price_fields: list[str] = ['high', 'low'],
|
||||
|
||||
) -> pl.DataFrame:
|
||||
'''
|
||||
Detect gaps in clearing price over an OHLC series.
|
||||
|
||||
2 types of gaps generally exist; up gaps and down gaps:
|
||||
|
||||
- UP gap: when any next sample's lo price is strictly greater
|
||||
then the current sample's hi price.
|
||||
|
||||
- DOWN gap: when any next sample's hi price is strictly
|
||||
less then the current samples lo price.
|
||||
|
||||
'''
|
||||
# return df.filter(
|
||||
# pl.col('high') - ) > expect_period,
|
||||
# ).select([
|
||||
# pl.dt.datetime(pl.col(time_col).shift(1)).suffix('_previous'),
|
||||
# pl.all(),
|
||||
# ]).select([
|
||||
# pl.all(),
|
||||
# (pl.col(time_col) - pl.col(f'{time_col}_previous')).alias('diff'),
|
||||
# ])
|
||||
...
|
||||
|
|
@ -26,9 +26,7 @@ from ..log import (
|
|||
)
|
||||
subsys: str = 'piker.data'
|
||||
|
||||
log = get_logger(
|
||||
name=subsys,
|
||||
)
|
||||
log = get_logger(subsys)
|
||||
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
|
|
|
|||
|
|
@ -27,15 +27,14 @@ from functools import partial
|
|||
from types import ModuleType
|
||||
from typing import (
|
||||
Any,
|
||||
Optional,
|
||||
Callable,
|
||||
AsyncContextManager,
|
||||
AsyncGenerator,
|
||||
Iterable,
|
||||
Type,
|
||||
)
|
||||
import json
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
from trio_websocket import (
|
||||
|
|
@ -68,7 +67,7 @@ class NoBsWs:
|
|||
|
||||
'''
|
||||
# apparently we can QoS for all sorts of reasons..so catch em.
|
||||
recon_errors: tuple[Type[Exception]] = (
|
||||
recon_errors = (
|
||||
ConnectionClosed,
|
||||
DisconnectionTimeout,
|
||||
ConnectionRejected,
|
||||
|
|
@ -106,10 +105,7 @@ class NoBsWs:
|
|||
def connected(self) -> bool:
|
||||
return self._connected.is_set()
|
||||
|
||||
async def reset(
|
||||
self,
|
||||
timeout: float,
|
||||
) -> bool:
|
||||
async def reset(self) -> None:
|
||||
'''
|
||||
Reset the underlying ws connection by cancelling
|
||||
the bg relay task and waiting for it to signal
|
||||
|
|
@ -118,31 +114,18 @@ class NoBsWs:
|
|||
'''
|
||||
self._connected = trio.Event()
|
||||
self._cs.cancel()
|
||||
with trio.move_on_after(timeout) as cs:
|
||||
await self._connected.wait()
|
||||
return True
|
||||
|
||||
assert cs.cancelled_caught
|
||||
return False
|
||||
await self._connected.wait()
|
||||
|
||||
async def send_msg(
|
||||
self,
|
||||
data: Any,
|
||||
timeout: float = 3,
|
||||
) -> None:
|
||||
while True:
|
||||
try:
|
||||
msg: Any = self._dumps(data)
|
||||
return await self._ws.send_message(msg)
|
||||
except self.recon_errors:
|
||||
with trio.CancelScope(shield=True):
|
||||
reconnected: bool = await self.reset(
|
||||
timeout=timeout,
|
||||
)
|
||||
if not reconnected:
|
||||
log.warning(
|
||||
'Failed to reconnect after {timeout!r}s ??'
|
||||
)
|
||||
await self.reset()
|
||||
|
||||
async def recv_msg(self) -> Any:
|
||||
msg: Any = await self._rx.receive()
|
||||
|
|
@ -184,7 +167,7 @@ async def _reconnect_forever(
|
|||
|
||||
async def proxy_msgs(
|
||||
ws: WebSocketConnection,
|
||||
rent_cs: trio.CancelScope, # parent cancel scope
|
||||
pcs: trio.CancelScope, # parent cancel scope
|
||||
):
|
||||
'''
|
||||
Receive (under `timeout` deadline) all msgs from from underlying
|
||||
|
|
@ -208,10 +191,8 @@ async def _reconnect_forever(
|
|||
f'{src_mod}\n'
|
||||
f'{url} connection bail with:'
|
||||
)
|
||||
with trio.CancelScope(shield=True):
|
||||
await trio.sleep(0.5)
|
||||
|
||||
rent_cs.cancel()
|
||||
await trio.sleep(0.5)
|
||||
pcs.cancel()
|
||||
|
||||
# go back to reonnect loop in parent task
|
||||
return
|
||||
|
|
@ -223,7 +204,7 @@ async def _reconnect_forever(
|
|||
f'{src_mod}\n'
|
||||
'WS feed seems down and slow af.. reconnecting\n'
|
||||
)
|
||||
rent_cs.cancel()
|
||||
pcs.cancel()
|
||||
|
||||
# go back to reonnect loop in parent task
|
||||
return
|
||||
|
|
@ -247,25 +228,16 @@ async def _reconnect_forever(
|
|||
nobsws._connected = trio.Event()
|
||||
task_status.started()
|
||||
|
||||
mc_state: trio._channel.MemoryChannelState = snd._state
|
||||
while (
|
||||
mc_state.open_receive_channels > 0
|
||||
and
|
||||
mc_state.open_send_channels > 0
|
||||
):
|
||||
while not snd._closed:
|
||||
log.info(
|
||||
f'{src_mod}\n'
|
||||
f'{url} trying (RE)CONNECT'
|
||||
)
|
||||
|
||||
ws: WebSocketConnection
|
||||
try:
|
||||
async with (
|
||||
open_websocket_url(url) as ws,
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
cs = nobsws._cs = tn.cancel_scope
|
||||
async with trio.open_nursery() as n:
|
||||
cs = nobsws._cs = n.cancel_scope
|
||||
ws: WebSocketConnection
|
||||
async with open_websocket_url(url) as ws:
|
||||
nobsws._ws = ws
|
||||
log.info(
|
||||
f'{src_mod}\n'
|
||||
|
|
@ -273,7 +245,7 @@ async def _reconnect_forever(
|
|||
)
|
||||
|
||||
# begin relay loop to forward msgs
|
||||
tn.start_soon(
|
||||
n.start_soon(
|
||||
proxy_msgs,
|
||||
ws,
|
||||
cs,
|
||||
|
|
@ -287,7 +259,7 @@ async def _reconnect_forever(
|
|||
|
||||
# TODO: should we return an explicit sub-cs
|
||||
# from this fixture task?
|
||||
await tn.start(
|
||||
await n.start(
|
||||
open_fixture,
|
||||
fixture,
|
||||
nobsws,
|
||||
|
|
@ -298,23 +270,8 @@ async def _reconnect_forever(
|
|||
nobsws._connected.set()
|
||||
await trio.sleep_forever()
|
||||
|
||||
except (
|
||||
HandshakeError,
|
||||
ConnectionRejected,
|
||||
):
|
||||
log.exception('Retrying connection')
|
||||
await trio.sleep(0.5) # throttle
|
||||
|
||||
except BaseException as _berr:
|
||||
berr = _berr
|
||||
log.exception(
|
||||
'Reconnect-attempt failed ??\n'
|
||||
)
|
||||
with trio.CancelScope(shield=True):
|
||||
await trio.sleep(0.2) # throttle
|
||||
raise berr
|
||||
|
||||
#|_ws & nursery block ends
|
||||
# ws open block end
|
||||
# nursery block end
|
||||
nobsws._connected = trio.Event()
|
||||
if cs.cancelled_caught:
|
||||
log.cancel(
|
||||
|
|
@ -327,8 +284,7 @@ async def _reconnect_forever(
|
|||
and not nobsws._connected.is_set()
|
||||
)
|
||||
|
||||
# -> from here, move to next reconnect attempt iteration
|
||||
# in the while loop above Bp
|
||||
# -> from here, move to next reconnect attempt
|
||||
|
||||
else:
|
||||
log.exception(
|
||||
|
|
@ -362,53 +318,43 @@ async def open_autorecon_ws(
|
|||
connetivity errors, or some user defined recv timeout.
|
||||
|
||||
You can provide a ``fixture`` async-context-manager which will be
|
||||
entered/exitted around each connection reset; eg. for
|
||||
(re)requesting subscriptions without requiring streaming setup
|
||||
code to rerun.
|
||||
entered/exitted around each connection reset; eg. for (re)requesting
|
||||
subscriptions without requiring streaming setup code to rerun.
|
||||
|
||||
'''
|
||||
snd: trio.MemorySendChannel
|
||||
rcv: trio.MemoryReceiveChannel
|
||||
snd, rcv = trio.open_memory_channel(616)
|
||||
|
||||
try:
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
nobsws = NoBsWs(
|
||||
url,
|
||||
rcv,
|
||||
msg_recv_timeout=msg_recv_timeout,
|
||||
)
|
||||
await tn.start(
|
||||
partial(
|
||||
_reconnect_forever,
|
||||
url,
|
||||
snd,
|
||||
nobsws,
|
||||
fixture=fixture,
|
||||
reset_after=reset_after,
|
||||
)
|
||||
)
|
||||
await nobsws._connected.wait()
|
||||
assert nobsws._cs
|
||||
assert nobsws.connected()
|
||||
try:
|
||||
yield nobsws
|
||||
finally:
|
||||
tn.cancel_scope.cancel()
|
||||
|
||||
except NoBsWs.recon_errors as con_err:
|
||||
log.warning(
|
||||
f'Entire ws-channel disconnect due to,\n'
|
||||
f'con_err: {con_err!r}\n'
|
||||
async with trio.open_nursery() as n:
|
||||
nobsws = NoBsWs(
|
||||
url,
|
||||
rcv,
|
||||
msg_recv_timeout=msg_recv_timeout,
|
||||
)
|
||||
await n.start(
|
||||
partial(
|
||||
_reconnect_forever,
|
||||
url,
|
||||
snd,
|
||||
nobsws,
|
||||
fixture=fixture,
|
||||
reset_after=reset_after,
|
||||
)
|
||||
)
|
||||
await nobsws._connected.wait()
|
||||
assert nobsws._cs
|
||||
assert nobsws.connected()
|
||||
|
||||
try:
|
||||
yield nobsws
|
||||
finally:
|
||||
n.cancel_scope.cancel()
|
||||
|
||||
|
||||
'''
|
||||
JSONRPC response-request style machinery for transparent multiplexing
|
||||
of msgs over a `NoBsWs`.
|
||||
JSONRPC response-request style machinery for transparent multiplexing of msgs
|
||||
over a NoBsWs.
|
||||
|
||||
'''
|
||||
|
||||
|
|
@ -416,8 +362,8 @@ of msgs over a `NoBsWs`.
|
|||
class JSONRPCResult(Struct):
|
||||
id: int
|
||||
jsonrpc: str = '2.0'
|
||||
result: dict|None = None
|
||||
error: dict|None = None
|
||||
result: Optional[dict] = None
|
||||
error: Optional[dict] = None
|
||||
|
||||
|
||||
@acm
|
||||
|
|
@ -425,82 +371,43 @@ async def open_jsonrpc_session(
|
|||
url: str,
|
||||
start_id: int = 0,
|
||||
response_type: type = JSONRPCResult,
|
||||
msg_recv_timeout: float = float('inf'),
|
||||
# ^NOTE, since only `deribit` is using this jsonrpc stuff atm
|
||||
# and options mkts are generally "slow moving"..
|
||||
#
|
||||
# FURTHER if we break the underlying ws connection then since we
|
||||
# don't pass a `fixture` to the task that manages `NoBsWs`, i.e.
|
||||
# `_reconnect_forever()`, the jsonrpc "transport pipe" get's
|
||||
# broken and never restored with wtv init sequence is required to
|
||||
# re-establish a working req-resp session.
|
||||
|
||||
request_type: Optional[type] = None,
|
||||
request_hook: Optional[Callable] = None,
|
||||
error_hook: Optional[Callable] = None,
|
||||
) -> Callable[[str, dict], dict]:
|
||||
'''
|
||||
Init a json-RPC-over-websocket connection to the provided `url`.
|
||||
|
||||
A `json_rpc: Callable[[str, dict], dict` is delivered to the
|
||||
caller for sending requests and a bg-`trio.Task` handles
|
||||
processing of response msgs including error reporting/raising in
|
||||
the parent/caller task.
|
||||
|
||||
'''
|
||||
# NOTE, store all request msgs so we can raise errors on the
|
||||
# caller side!
|
||||
req_msgs: dict[int, dict] = {}
|
||||
|
||||
async with (
|
||||
trio.open_nursery() as tn,
|
||||
open_autorecon_ws(
|
||||
url=url,
|
||||
msg_recv_timeout=msg_recv_timeout,
|
||||
) as ws
|
||||
trio.open_nursery() as n,
|
||||
open_autorecon_ws(url) as ws
|
||||
):
|
||||
rpc_id: Iterable[int] = count(start_id)
|
||||
rpc_id: Iterable = count(start_id)
|
||||
rpc_results: dict[int, dict] = {}
|
||||
|
||||
async def json_rpc(
|
||||
method: str,
|
||||
params: dict,
|
||||
) -> dict:
|
||||
async def json_rpc(method: str, params: dict) -> dict:
|
||||
'''
|
||||
perform a json rpc call and wait for the result, raise exception in
|
||||
case of error field present on response
|
||||
'''
|
||||
nonlocal req_msgs
|
||||
|
||||
req_id: int = next(rpc_id)
|
||||
msg = {
|
||||
'jsonrpc': '2.0',
|
||||
'id': req_id,
|
||||
'id': next(rpc_id),
|
||||
'method': method,
|
||||
'params': params
|
||||
}
|
||||
_id = msg['id']
|
||||
|
||||
result = rpc_results[_id] = {
|
||||
rpc_results[_id] = {
|
||||
'result': None,
|
||||
'error': None,
|
||||
'event': trio.Event(), # signal caller resp arrived
|
||||
'event': trio.Event()
|
||||
}
|
||||
req_msgs[_id] = msg
|
||||
|
||||
await ws.send_msg(msg)
|
||||
|
||||
# wait for reponse before unblocking requester code
|
||||
await rpc_results[_id]['event'].wait()
|
||||
|
||||
if (maybe_result := result['result']):
|
||||
ret = maybe_result
|
||||
del rpc_results[_id]
|
||||
ret = rpc_results[_id]['result']
|
||||
|
||||
else:
|
||||
err = result['error']
|
||||
raise Exception(
|
||||
f'JSONRPC request failed\n'
|
||||
f'req: {msg}\n'
|
||||
f'resp: {err}\n'
|
||||
)
|
||||
del rpc_results[_id]
|
||||
|
||||
if ret.error is not None:
|
||||
raise Exception(json.dumps(ret.error, indent=4))
|
||||
|
|
@ -515,7 +422,6 @@ async def open_jsonrpc_session(
|
|||
the server side.
|
||||
|
||||
'''
|
||||
nonlocal req_msgs
|
||||
async for msg in ws:
|
||||
match msg:
|
||||
case {
|
||||
|
|
@ -539,28 +445,19 @@ async def open_jsonrpc_session(
|
|||
'params': _,
|
||||
}:
|
||||
log.debug(f'Recieved\n{msg}')
|
||||
if request_hook:
|
||||
await request_hook(request_type(**msg))
|
||||
|
||||
case {
|
||||
'error': error
|
||||
}:
|
||||
# retreive orig request msg, set error
|
||||
# response in original "result" msg,
|
||||
# THEN FINALLY set the event to signal caller
|
||||
# to raise the error in the parent task.
|
||||
req_id: int = error['id']
|
||||
req_msg: dict = req_msgs[req_id]
|
||||
result: dict = rpc_results[req_id]
|
||||
result['error'] = error
|
||||
result['event'].set()
|
||||
log.error(
|
||||
f'JSONRPC request failed\n'
|
||||
f'req: {req_msg}\n'
|
||||
f'resp: {error}\n'
|
||||
)
|
||||
log.warning(f'Recieved\n{error}')
|
||||
if error_hook:
|
||||
await error_hook(response_type(**msg))
|
||||
|
||||
case _:
|
||||
log.warning(f'Unhandled JSON-RPC msg!?\n{msg}')
|
||||
|
||||
tn.start_soon(recv_task)
|
||||
n.start_soon(recv_task)
|
||||
yield json_rpc
|
||||
tn.cancel_scope.cancel()
|
||||
n.cancel_scope.cancel()
|
||||
|
|
|
|||
|
|
@ -28,7 +28,6 @@ module.
|
|||
from __future__ import annotations
|
||||
from collections import (
|
||||
defaultdict,
|
||||
abc,
|
||||
)
|
||||
from contextlib import asynccontextmanager as acm
|
||||
from functools import partial
|
||||
|
|
@ -37,16 +36,19 @@ from types import ModuleType
|
|||
from typing import (
|
||||
Any,
|
||||
AsyncContextManager,
|
||||
Optional,
|
||||
Awaitable,
|
||||
Sequence,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import trio
|
||||
from trio.abc import ReceiveChannel
|
||||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from tractor import trionics
|
||||
from tractor.trionics import (
|
||||
maybe_open_context,
|
||||
gather_contexts,
|
||||
)
|
||||
|
||||
from piker.accounting import (
|
||||
MktPair,
|
||||
|
|
@ -57,16 +59,18 @@ from piker.brokers import get_brokermod
|
|||
from piker.service import (
|
||||
maybe_spawn_brokerd,
|
||||
)
|
||||
from piker.ui import _search
|
||||
from piker.calc import humanize
|
||||
from ._util import (
|
||||
log,
|
||||
get_console_log,
|
||||
)
|
||||
from .flows import Flume
|
||||
from .validate import (
|
||||
FeedInit,
|
||||
validate_backend,
|
||||
)
|
||||
from ..tsp import (
|
||||
from .history import (
|
||||
manage_history,
|
||||
)
|
||||
from .ingest import get_ingestormod
|
||||
|
|
@ -75,36 +79,6 @@ from ._sampling import (
|
|||
uniform_rate_send,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .flows import Flume
|
||||
from tractor._addr import Address
|
||||
from tractor.msg.types import Aid
|
||||
|
||||
|
||||
class Sub(Struct, frozen=True):
|
||||
'''
|
||||
A live feed subscription entry.
|
||||
|
||||
Contains meta-data on the remote-actor type (in functionality
|
||||
terms) as well as refs to IPC streams and sampler runtime
|
||||
params.
|
||||
|
||||
'''
|
||||
ipc: tractor.MsgStream
|
||||
send_chan: trio.abc.SendChannel | None = None
|
||||
|
||||
# tick throttle rate in Hz; determines how live
|
||||
# quotes/ticks should be downsampled before relay
|
||||
# to the receiving remote consumer (process).
|
||||
throttle_rate: float | None = None
|
||||
_throttle_cs: trio.CancelScope | None = None
|
||||
|
||||
# TODO: actually stash comms info for the far end to allow
|
||||
# `.tsp`, `.fsp` and `.data._sampling` sub-systems to re-render
|
||||
# the data view as needed via msging with the `._remote_ctl`
|
||||
# ipc ctx.
|
||||
rc_ui: bool = False
|
||||
|
||||
|
||||
class _FeedsBus(Struct):
|
||||
'''
|
||||
|
|
@ -130,7 +104,13 @@ class _FeedsBus(Struct):
|
|||
|
||||
_subscribers: defaultdict[
|
||||
str,
|
||||
set[Sub]
|
||||
set[
|
||||
tuple[
|
||||
tractor.MsgStream | trio.MemorySendChannel,
|
||||
# tractor.Context,
|
||||
float | None, # tick throttle in Hz
|
||||
]
|
||||
]
|
||||
] = defaultdict(set)
|
||||
|
||||
async def start_task(
|
||||
|
|
@ -145,8 +125,6 @@ class _FeedsBus(Struct):
|
|||
trio.CancelScope] = trio.TASK_STATUS_IGNORED,
|
||||
) -> None:
|
||||
with trio.CancelScope() as cs:
|
||||
# TODO: shouldn't this be a direct await to avoid
|
||||
# cancellation contagion to the bus nursery!?!?!
|
||||
await self.nursery.start(
|
||||
target,
|
||||
*args,
|
||||
|
|
@ -164,28 +142,31 @@ class _FeedsBus(Struct):
|
|||
def get_subs(
|
||||
self,
|
||||
key: str,
|
||||
|
||||
) -> set[Sub]:
|
||||
) -> set[
|
||||
tuple[
|
||||
tractor.MsgStream | trio.MemorySendChannel,
|
||||
float | None, # tick throttle in Hz
|
||||
]
|
||||
]:
|
||||
'''
|
||||
Get the ``set`` of consumer subscription entries for the given key.
|
||||
|
||||
'''
|
||||
return self._subscribers[key]
|
||||
|
||||
def subs_items(self) -> abc.ItemsView[str, set[Sub]]:
|
||||
return self._subscribers.items()
|
||||
|
||||
def add_subs(
|
||||
self,
|
||||
key: str,
|
||||
subs: set[Sub],
|
||||
|
||||
) -> set[Sub]:
|
||||
subs: set[tuple[
|
||||
tractor.MsgStream | trio.MemorySendChannel,
|
||||
float | None, # tick throttle in Hz
|
||||
]],
|
||||
) -> set[tuple]:
|
||||
'''
|
||||
Add a ``set`` of consumer subscription entries for the given key.
|
||||
|
||||
'''
|
||||
_subs: set[Sub] = self._subscribers.setdefault(key, set())
|
||||
_subs: set[tuple] = self._subscribers[key]
|
||||
_subs.update(subs)
|
||||
return _subs
|
||||
|
||||
|
|
@ -239,6 +220,7 @@ async def allocate_persistent_feed(
|
|||
|
||||
brokername: str,
|
||||
symstr: str,
|
||||
|
||||
loglevel: str,
|
||||
start_stream: bool = True,
|
||||
init_timeout: float = 616,
|
||||
|
|
@ -277,7 +259,7 @@ async def allocate_persistent_feed(
|
|||
# ``stream_quotes()``, a required broker backend endpoint.
|
||||
init_msgs: (
|
||||
list[FeedInit] # new
|
||||
|dict[str, dict[str, str]] # legacy / deprecated
|
||||
| dict[str, dict[str, str]] # legacy / deprecated
|
||||
)
|
||||
|
||||
# TODO: probably make a struct msg type for this as well
|
||||
|
|
@ -347,25 +329,19 @@ async def allocate_persistent_feed(
|
|||
izero_rt,
|
||||
rt_shm,
|
||||
) = await bus.nursery.start(
|
||||
partial(
|
||||
manage_history,
|
||||
mod=mod,
|
||||
mkt=mkt,
|
||||
some_data_ready=some_data_ready,
|
||||
feed_is_live=feed_is_live,
|
||||
loglevel=loglevel,
|
||||
)
|
||||
manage_history,
|
||||
mod,
|
||||
bus,
|
||||
mkt,
|
||||
some_data_ready,
|
||||
feed_is_live,
|
||||
)
|
||||
|
||||
# yield back control to starting nursery once we receive either
|
||||
# some history or a real-time quote.
|
||||
log.info(
|
||||
f'loading OHLCV history: {fqme!r}\n'
|
||||
)
|
||||
log.info(f'loading OHLCV history: {fqme}')
|
||||
await some_data_ready.wait()
|
||||
|
||||
# XXX, avoid cycle; it imports this mod.
|
||||
from .flows import Flume
|
||||
flume = Flume(
|
||||
|
||||
# TODO: we have to use this for now since currently the
|
||||
|
|
@ -432,13 +408,7 @@ async def allocate_persistent_feed(
|
|||
rt_shm.array['time'][1] = ts + 1
|
||||
|
||||
elif hist_shm.array.size == 0:
|
||||
for i in range(100):
|
||||
await trio.sleep(0.1)
|
||||
if hist_shm.array.size > 0:
|
||||
break
|
||||
else:
|
||||
await tractor.pause()
|
||||
raise RuntimeError(f'History (1m) Shm for {fqme} is empty!?')
|
||||
raise RuntimeError(f'History (1m) Shm for {fqme} is empty!?')
|
||||
|
||||
# wait the spawning parent task to register its subscriber
|
||||
# send-stream entry before we start the sample loop.
|
||||
|
|
@ -462,14 +432,14 @@ async def allocate_persistent_feed(
|
|||
|
||||
@tractor.context
|
||||
async def open_feed_bus(
|
||||
|
||||
ctx: tractor.Context,
|
||||
brokername: str,
|
||||
symbols: list[str], # normally expected to the broker-specific fqme
|
||||
|
||||
loglevel: str = 'error',
|
||||
tick_throttle: float | None = None,
|
||||
tick_throttle: Optional[float] = None,
|
||||
start_stream: bool = True,
|
||||
allow_remote_ctl_ui: bool = False,
|
||||
|
||||
) -> dict[
|
||||
str, # fqme
|
||||
|
|
@ -482,17 +452,10 @@ async def open_feed_bus(
|
|||
|
||||
'''
|
||||
if loglevel is None:
|
||||
loglevel: str = tractor.current_actor().loglevel
|
||||
loglevel = tractor.current_actor().loglevel
|
||||
|
||||
# XXX: required to propagate ``tractor`` loglevel to piker
|
||||
# logging
|
||||
get_console_log(
|
||||
level=(loglevel
|
||||
or
|
||||
tractor.current_actor().loglevel
|
||||
),
|
||||
name=__name__,
|
||||
)
|
||||
# XXX: required to propagate ``tractor`` loglevel to piker logging
|
||||
get_console_log(loglevel or tractor.current_actor().loglevel)
|
||||
|
||||
# local state sanity checks
|
||||
# TODO: check for any stale shm entries for this symbol
|
||||
|
|
@ -502,10 +465,11 @@ async def open_feed_bus(
|
|||
assert 'brokerd' in servicename
|
||||
assert brokername in servicename
|
||||
|
||||
bus: _FeedsBus = get_feed_bus(brokername)
|
||||
bus = get_feed_bus(brokername)
|
||||
sub_registered = trio.Event()
|
||||
|
||||
flumes: dict[str, Flume] = {}
|
||||
|
||||
for symbol in symbols:
|
||||
|
||||
# if no cached feed for this symbol has been created for this
|
||||
|
|
@ -548,10 +512,10 @@ async def open_feed_bus(
|
|||
# pack for ``.started()`` sync msg
|
||||
flumes[fqme] = flume
|
||||
|
||||
# we use the broker-specific fqme (bs_fqme) for the sampler
|
||||
# subscription since the backend isn't (yet) expected to
|
||||
# append it's own name to the fqme, so we filter on keys
|
||||
# which *do not* include that name (e.g .ib) .
|
||||
# we use the broker-specific fqme (bs_fqme) for the
|
||||
# sampler subscription since the backend isn't (yet) expected to
|
||||
# append it's own name to the fqme, so we filter on keys which
|
||||
# *do not* include that name (e.g .ib) .
|
||||
bus._subscribers.setdefault(bs_fqme, set())
|
||||
|
||||
# sync feed subscribers with flume handles
|
||||
|
|
@ -590,60 +554,49 @@ async def open_feed_bus(
|
|||
# that the ``sample_and_broadcast()`` task (spawned inside
|
||||
# ``allocate_persistent_feed()``) will push real-time quote
|
||||
# (ticks) to this new consumer.
|
||||
cs: trio.CancelScope | None = None
|
||||
send: trio.MemorySendChannel | None = None
|
||||
|
||||
if tick_throttle:
|
||||
flume.throttle_rate = tick_throttle
|
||||
|
||||
# open a bg task which receives quotes over a mem
|
||||
# chan and only pushes them to the target
|
||||
# actor-consumer at a max ``tick_throttle``
|
||||
# (instantaneous) rate.
|
||||
# open a bg task which receives quotes over a mem chan
|
||||
# and only pushes them to the target actor-consumer at
|
||||
# a max ``tick_throttle`` instantaneous rate.
|
||||
send, recv = trio.open_memory_channel(2**10)
|
||||
|
||||
# NOTE: the ``.send`` channel here is a swapped-in
|
||||
# trio mem chan which gets `.send()`-ed by the normal
|
||||
# sampler task but instead of being sent directly
|
||||
# over the IPC msg stream it's the throttle task
|
||||
# does the work of incrementally forwarding to the
|
||||
# IPC stream at the throttle rate.
|
||||
cs: trio.CancelScope = await bus.start_task(
|
||||
cs = await bus.start_task(
|
||||
uniform_rate_send,
|
||||
tick_throttle,
|
||||
recv,
|
||||
stream,
|
||||
)
|
||||
# NOTE: so the ``send`` channel here is actually a swapped
|
||||
# in trio mem chan which gets pushed by the normal sampler
|
||||
# task but instead of being sent directly over the IPC msg
|
||||
# stream it's the throttle task does the work of
|
||||
# incrementally forwarding to the IPC stream at the throttle
|
||||
# rate.
|
||||
send._ctx = ctx # mock internal ``tractor.MsgStream`` ref
|
||||
sub = (send, tick_throttle)
|
||||
|
||||
sub = Sub(
|
||||
ipc=stream,
|
||||
send_chan=send,
|
||||
throttle_rate=tick_throttle,
|
||||
_throttle_cs=cs,
|
||||
rc_ui=allow_remote_ctl_ui,
|
||||
)
|
||||
else:
|
||||
sub = (stream, tick_throttle)
|
||||
|
||||
# TODO: add an api for this on the bus?
|
||||
# maybe use the current task-id to key the sub list that's
|
||||
# added / removed? Or maybe we can add a general
|
||||
# pause-resume by sub-key api?
|
||||
bs_fqme = fqme.removesuffix(f'.{brokername}')
|
||||
local_subs.setdefault(
|
||||
bs_fqme,
|
||||
set()
|
||||
).add(sub)
|
||||
bus.add_subs(
|
||||
bs_fqme,
|
||||
{sub}
|
||||
)
|
||||
local_subs.setdefault(bs_fqme, set()).add(sub)
|
||||
bus.add_subs(bs_fqme, {sub})
|
||||
|
||||
# sync caller with all subs registered state
|
||||
sub_registered.set()
|
||||
|
||||
uid: tuple[str, str] = ctx.chan.uid
|
||||
uid = ctx.chan.uid
|
||||
try:
|
||||
# ctrl protocol for start/stop of live quote streams
|
||||
# based on UI state (eg. don't need a stream when
|
||||
# a symbol isn't being displayed).
|
||||
# ctrl protocol for start/stop of quote streams based on UI
|
||||
# state (eg. don't need a stream when a symbol isn't being
|
||||
# displayed).
|
||||
async for msg in stream:
|
||||
|
||||
if msg == 'pause':
|
||||
|
|
@ -689,7 +642,6 @@ class Feed(Struct):
|
|||
'''
|
||||
mods: dict[str, ModuleType] = {}
|
||||
portals: dict[ModuleType, tractor.Portal] = {}
|
||||
|
||||
flumes: dict[
|
||||
str, # FQME
|
||||
Flume,
|
||||
|
|
@ -736,10 +688,7 @@ class Feed(Struct):
|
|||
async for msg in stream:
|
||||
await tx.send(msg)
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as nurse
|
||||
):
|
||||
async with trio.open_nursery() as nurse:
|
||||
# spawn a relay task for each stream so that they all
|
||||
# multiplex to a common channel.
|
||||
for brokername in mods:
|
||||
|
|
@ -785,7 +734,6 @@ async def install_brokerd_search(
|
|||
except trio.EndOfChannel:
|
||||
return {}
|
||||
|
||||
from piker.ui import _search
|
||||
async with _search.register_symbol_search(
|
||||
|
||||
provider_name=brokermod.name,
|
||||
|
|
@ -802,8 +750,9 @@ async def install_brokerd_search(
|
|||
|
||||
@acm
|
||||
async def maybe_open_feed(
|
||||
|
||||
fqmes: list[str],
|
||||
loglevel: str|None = None,
|
||||
loglevel: Optional[str] = None,
|
||||
|
||||
**kwargs,
|
||||
|
||||
|
|
@ -819,7 +768,7 @@ async def maybe_open_feed(
|
|||
'''
|
||||
fqme = fqmes[0]
|
||||
|
||||
async with trionics.maybe_open_context(
|
||||
async with maybe_open_context(
|
||||
acm_func=open_feed,
|
||||
kwargs={
|
||||
'fqmes': fqmes,
|
||||
|
|
@ -839,7 +788,7 @@ async def maybe_open_feed(
|
|||
# add a new broadcast subscription for the quote stream
|
||||
# if this feed is likely already in use
|
||||
|
||||
async with trionics.gather_contexts(
|
||||
async with gather_contexts(
|
||||
mngrs=[stream.subscribe() for stream in feed.streams.values()]
|
||||
) as bstreams:
|
||||
for bstream, flume in zip(bstreams, feed.flumes.values()):
|
||||
|
|
@ -855,14 +804,13 @@ async def maybe_open_feed(
|
|||
|
||||
@acm
|
||||
async def open_feed(
|
||||
|
||||
fqmes: list[str],
|
||||
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
allow_overruns: bool = True,
|
||||
start_stream: bool = True,
|
||||
tick_throttle: float|None = None, # Hz
|
||||
|
||||
allow_remote_ctl_ui: bool = False,
|
||||
tick_throttle: float | None = None, # Hz
|
||||
|
||||
) -> Feed:
|
||||
'''
|
||||
|
|
@ -887,6 +835,7 @@ async def open_feed(
|
|||
|
||||
# one actor per brokerd for now
|
||||
brokerd_ctxs = []
|
||||
|
||||
for brokermod, bfqmes in providers.items():
|
||||
|
||||
# if no `brokerd` for this backend exists yet we spawn
|
||||
|
|
@ -899,7 +848,7 @@ async def open_feed(
|
|||
)
|
||||
|
||||
portals: tuple[tractor.Portal]
|
||||
async with trionics.gather_contexts(
|
||||
async with gather_contexts(
|
||||
brokerd_ctxs,
|
||||
) as portals:
|
||||
|
||||
|
|
@ -912,19 +861,19 @@ async def open_feed(
|
|||
feed.portals[brokermod] = portal
|
||||
|
||||
# fill out "status info" that the UI can show
|
||||
chan: tractor.Channel = portal.chan
|
||||
raddr: Address = chan.raddr
|
||||
aid: Aid = chan.aid
|
||||
# TAG_feed_status_update
|
||||
host, port = portal.channel.raddr
|
||||
if host == '127.0.0.1':
|
||||
host = 'localhost'
|
||||
|
||||
feed.status.update({
|
||||
'actor_id': aid,
|
||||
'actor_short_id': f'{aid.name}@{aid.pid}',
|
||||
'ipc': chan.raddr.proto_key,
|
||||
'ipc_addr': raddr,
|
||||
'actor_name': portal.channel.uid[0],
|
||||
'host': host,
|
||||
'port': port,
|
||||
'hist_shm': 'NA',
|
||||
'rt_shm': 'NA',
|
||||
'throttle_hz': tick_throttle,
|
||||
'throttle_rate': tick_throttle,
|
||||
})
|
||||
# feed.status.update(init_msg.pop('status', {}))
|
||||
|
||||
# (allocate and) connect to any feed bus for this broker
|
||||
bus_ctxs.append(
|
||||
|
|
@ -945,21 +894,13 @@ async def open_feed(
|
|||
# of these stream open sequences sequentially per
|
||||
# backend? .. need some thot!
|
||||
allow_overruns=True,
|
||||
|
||||
# NOTE: UI actors (like charts) can allow
|
||||
# remote control of certain graphics rendering
|
||||
# capabilities via the
|
||||
# `.ui._remote_ctl.remote_annotate()` msg loop.
|
||||
allow_remote_ctl_ui=allow_remote_ctl_ui,
|
||||
)
|
||||
)
|
||||
|
||||
assert len(feed.mods) == len(feed.portals)
|
||||
|
||||
# XXX, avoid cycle; it imports this mod.
|
||||
from .flows import Flume
|
||||
async with (
|
||||
trionics.gather_contexts(bus_ctxs) as ctxs,
|
||||
gather_contexts(bus_ctxs) as ctxs,
|
||||
):
|
||||
stream_ctxs: list[tractor.MsgStream] = []
|
||||
for (
|
||||
|
|
@ -1001,7 +942,7 @@ async def open_feed(
|
|||
brokermod: ModuleType
|
||||
fqmes: list[str]
|
||||
async with (
|
||||
trionics.gather_contexts(stream_ctxs) as streams,
|
||||
gather_contexts(stream_ctxs) as streams,
|
||||
):
|
||||
for (
|
||||
stream,
|
||||
|
|
@ -1017,12 +958,6 @@ async def open_feed(
|
|||
if brokermod.name == flume.mkt.broker:
|
||||
flume.stream = stream
|
||||
|
||||
assert (
|
||||
len(feed.mods)
|
||||
==
|
||||
len(feed.portals)
|
||||
==
|
||||
len(feed.streams)
|
||||
)
|
||||
assert len(feed.mods) == len(feed.portals) == len(feed.streams)
|
||||
|
||||
yield feed
|
||||
|
|
|
|||
|
|
@ -36,21 +36,41 @@ from ._sharedmem import (
|
|||
ShmArray,
|
||||
_Token,
|
||||
)
|
||||
from piker.accounting import MktPair
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.data.feed import Feed
|
||||
from ..accounting import MktPair
|
||||
from .feed import Feed
|
||||
|
||||
|
||||
# TODO: ideas for further abstractions as per
|
||||
# https://github.com/pikers/piker/issues/216 and
|
||||
# https://github.com/pikers/piker/issues/270:
|
||||
# - a ``Cascade`` would be the minimal "connection" of 2 ``Flumes``
|
||||
# as per circuit parlance:
|
||||
# https://en.wikipedia.org/wiki/Two-port_network#Cascade_connection
|
||||
# - could cover the combination of our `FspAdmin` and the
|
||||
# backend `.fsp._engine` related machinery to "connect" one flume
|
||||
# to another?
|
||||
# - a (financial signal) ``Flow`` would be the a "collection" of such
|
||||
# minmial cascades. Some engineering based jargon concepts:
|
||||
# - https://en.wikipedia.org/wiki/Signal_chain
|
||||
# - https://en.wikipedia.org/wiki/Daisy_chain_(electrical_engineering)
|
||||
# - https://en.wikipedia.org/wiki/Audio_signal_flow
|
||||
# - https://en.wikipedia.org/wiki/Digital_signal_processing#Implementation
|
||||
# - https://en.wikipedia.org/wiki/Dataflow_programming
|
||||
# - https://en.wikipedia.org/wiki/Signal_programming
|
||||
# - https://en.wikipedia.org/wiki/Incremental_computing
|
||||
|
||||
|
||||
class Flume(Struct):
|
||||
'''
|
||||
Composite reference type which points to all the addressing
|
||||
handles and other meta-data necessary for the read, measure and
|
||||
management of a set of real-time updated data flows.
|
||||
Composite reference type which points to all the addressing handles
|
||||
and other meta-data necessary for the read, measure and management
|
||||
of a set of real-time updated data flows.
|
||||
|
||||
Can be thought of as a "flow descriptor" or "flow frame" which
|
||||
describes the high level properties of a set of data flows that
|
||||
can be used seamlessly across process-memory boundaries.
|
||||
describes the high level properties of a set of data flows that can
|
||||
be used seamlessly across process-memory boundaries.
|
||||
|
||||
Each instance's sub-components normally includes:
|
||||
- a msg oriented quote stream provided via an IPC transport
|
||||
|
|
@ -73,7 +93,6 @@ class Flume(Struct):
|
|||
# private shm refs loaded dynamically from tokens
|
||||
_hist_shm: ShmArray | None = None
|
||||
_rt_shm: ShmArray | None = None
|
||||
_readonly: bool = True
|
||||
|
||||
stream: tractor.MsgStream | None = None
|
||||
izero_hist: int = 0
|
||||
|
|
@ -82,7 +101,7 @@ class Flume(Struct):
|
|||
|
||||
# TODO: do we need this really if we can pull the `Portal` from
|
||||
# ``tractor``'s internals?
|
||||
feed: Feed|None = None
|
||||
feed: Feed | None = None
|
||||
|
||||
@property
|
||||
def rt_shm(self) -> ShmArray:
|
||||
|
|
@ -90,7 +109,7 @@ class Flume(Struct):
|
|||
if self._rt_shm is None:
|
||||
self._rt_shm = attach_shm_array(
|
||||
token=self._rt_shm_token,
|
||||
readonly=self._readonly,
|
||||
readonly=True,
|
||||
)
|
||||
|
||||
return self._rt_shm
|
||||
|
|
@ -103,10 +122,12 @@ class Flume(Struct):
|
|||
'No shm token has been set for the history buffer?'
|
||||
)
|
||||
|
||||
if self._hist_shm is None:
|
||||
if (
|
||||
self._hist_shm is None
|
||||
):
|
||||
self._hist_shm = attach_shm_array(
|
||||
token=self._hist_shm_token,
|
||||
readonly=self._readonly,
|
||||
readonly=True,
|
||||
)
|
||||
|
||||
return self._hist_shm
|
||||
|
|
@ -125,10 +146,10 @@ class Flume(Struct):
|
|||
period and ratio between them.
|
||||
|
||||
'''
|
||||
times: np.ndarray = self.hist_shm.array['time']
|
||||
end: float | int = pendulum.from_timestamp(times[-1])
|
||||
start: float | int = pendulum.from_timestamp(times[times != times[-1]][-1])
|
||||
hist_step_size_s: float = (end - start).seconds
|
||||
times = self.hist_shm.array['time']
|
||||
end = pendulum.from_timestamp(times[-1])
|
||||
start = pendulum.from_timestamp(times[times != times[-1]][-1])
|
||||
hist_step_size_s = (end - start).seconds
|
||||
|
||||
times = self.rt_shm.array['time']
|
||||
end = pendulum.from_timestamp(times[-1])
|
||||
|
|
@ -148,25 +169,17 @@ class Flume(Struct):
|
|||
msg = self.to_dict()
|
||||
msg['mkt'] = self.mkt.to_dict()
|
||||
|
||||
# NOTE: pop all un-msg-serializable fields:
|
||||
# - `tractor.MsgStream`
|
||||
# - `Feed`
|
||||
# - `Shmarray`
|
||||
# it's expected the `.from_msg()` on the other side
|
||||
# will get instead some kind of msg-compat version
|
||||
# that it can load.
|
||||
# can't serialize the stream or feed objects, it's expected
|
||||
# you'll have a ref to it since this msg should be rxed on
|
||||
# a stream on whatever far end IPC..
|
||||
msg.pop('stream')
|
||||
msg.pop('feed')
|
||||
msg.pop('_rt_shm')
|
||||
msg.pop('_hist_shm')
|
||||
|
||||
return msg
|
||||
|
||||
@classmethod
|
||||
def from_msg(
|
||||
cls,
|
||||
msg: dict,
|
||||
readonly: bool = True,
|
||||
|
||||
) -> dict:
|
||||
'''
|
||||
|
|
@ -177,11 +190,7 @@ class Flume(Struct):
|
|||
mkt_msg = msg.pop('mkt')
|
||||
from ..accounting import MktPair # cycle otherwise..
|
||||
mkt = MktPair.from_msg(mkt_msg)
|
||||
msg |= {'_readonly': readonly}
|
||||
return cls(
|
||||
mkt=mkt,
|
||||
**msg,
|
||||
)
|
||||
return cls(mkt=mkt, **msg)
|
||||
|
||||
def get_index(
|
||||
self,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,982 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) Tyler Goodlet (in stewardship for pikers)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Affero General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
Historical data business logic for load, backfill and tsdb storage.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
# from collections import (
|
||||
# Counter,
|
||||
# )
|
||||
from datetime import datetime
|
||||
from functools import partial
|
||||
# import time
|
||||
from types import ModuleType
|
||||
from typing import (
|
||||
Callable,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from pendulum import (
|
||||
Duration,
|
||||
from_timestamp,
|
||||
)
|
||||
import numpy as np
|
||||
|
||||
from ..accounting import (
|
||||
MktPair,
|
||||
)
|
||||
from ._util import (
|
||||
log,
|
||||
)
|
||||
from ._sharedmem import (
|
||||
maybe_open_shm_array,
|
||||
ShmArray,
|
||||
)
|
||||
from ._source import def_iohlcv_fields
|
||||
from ._sampling import (
|
||||
open_sample_stream,
|
||||
)
|
||||
from ..brokers._util import (
|
||||
DataUnavailable,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from bidict import bidict
|
||||
from ..service.marketstore import StorageClient
|
||||
from .feed import _FeedsBus
|
||||
|
||||
|
||||
# `ShmArray` buffer sizing configuration:
|
||||
_mins_in_day = int(60 * 24)
|
||||
# how much is probably dependent on lifestyle
|
||||
# but we reco a buncha times (but only on a
|
||||
# run-every-other-day kinda week).
|
||||
_secs_in_day = int(60 * _mins_in_day)
|
||||
_days_in_week: int = 7
|
||||
|
||||
_days_worth: int = 3
|
||||
_default_hist_size: int = 6 * 365 * _mins_in_day
|
||||
_hist_buffer_start = int(
|
||||
_default_hist_size - round(7 * _mins_in_day)
|
||||
)
|
||||
|
||||
_default_rt_size: int = _days_worth * _secs_in_day
|
||||
# NOTE: start the append index in rt buffer such that 1 day's worth
|
||||
# can be appenened before overrun.
|
||||
_rt_buffer_start = int((_days_worth - 1) * _secs_in_day)
|
||||
|
||||
|
||||
def diff_history(
|
||||
array: np.ndarray,
|
||||
append_until_dt: datetime | None = None,
|
||||
prepend_until_dt: datetime | None = None,
|
||||
|
||||
) -> np.ndarray:
|
||||
|
||||
# no diffing with tsdb dt index possible..
|
||||
if (
|
||||
prepend_until_dt is None
|
||||
and append_until_dt is None
|
||||
):
|
||||
return array
|
||||
|
||||
times = array['time']
|
||||
|
||||
if append_until_dt:
|
||||
return array[times < append_until_dt.timestamp()]
|
||||
else:
|
||||
return array[times >= prepend_until_dt.timestamp()]
|
||||
|
||||
|
||||
async def shm_push_in_between(
|
||||
shm: ShmArray,
|
||||
to_push: np.ndarray,
|
||||
prepend_index: int,
|
||||
|
||||
update_start_on_prepend: bool = False,
|
||||
|
||||
) -> int:
|
||||
shm.push(
|
||||
to_push,
|
||||
prepend=True,
|
||||
|
||||
# XXX: only update the ._first index if no tsdb
|
||||
# segment was previously prepended by the
|
||||
# parent task.
|
||||
update_first=update_start_on_prepend,
|
||||
|
||||
# XXX: only prepend from a manually calculated shm
|
||||
# index if there was already a tsdb history
|
||||
# segment prepended (since then the
|
||||
# ._first.value is going to be wayyy in the
|
||||
# past!)
|
||||
start=(
|
||||
prepend_index
|
||||
if not update_start_on_prepend
|
||||
else None
|
||||
),
|
||||
)
|
||||
# XXX: extremely important, there can be no checkpoints
|
||||
# in the block above to avoid entering new ``frames``
|
||||
# values while we're pipelining the current ones to
|
||||
# memory...
|
||||
array = shm.array
|
||||
zeros = array[array['low'] == 0]
|
||||
|
||||
# always backfill gaps with the earliest (price) datum's
|
||||
# value to avoid the y-ranger including zeros and completely
|
||||
# stretching the y-axis..
|
||||
if 0 < zeros.size:
|
||||
zeros[[
|
||||
'open',
|
||||
'high',
|
||||
'low',
|
||||
'close',
|
||||
]] = shm._array[zeros['index'][0] - 1]['close']
|
||||
# await tractor.pause()
|
||||
|
||||
|
||||
async def start_backfill(
|
||||
get_hist,
|
||||
mod: ModuleType,
|
||||
mkt: MktPair,
|
||||
shm: ShmArray,
|
||||
timeframe: float,
|
||||
|
||||
backfill_from_shm_index: int,
|
||||
backfill_from_dt: datetime,
|
||||
|
||||
sampler_stream: tractor.MsgStream,
|
||||
|
||||
backfill_until_dt: datetime | None = None,
|
||||
storage: StorageClient | None = None,
|
||||
|
||||
write_tsdb: bool = True,
|
||||
|
||||
task_status: TaskStatus[tuple] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> int:
|
||||
|
||||
# let caller unblock and deliver latest history frame
|
||||
# and use to signal that backfilling the shm gap until
|
||||
# the tsdb end is complete!
|
||||
bf_done = trio.Event()
|
||||
task_status.started(bf_done)
|
||||
|
||||
# based on the sample step size, maybe load a certain amount history
|
||||
update_start_on_prepend: bool = False
|
||||
if backfill_until_dt is None:
|
||||
|
||||
# TODO: drop this right and just expose the backfill
|
||||
# limits inside a [storage] section in conf.toml?
|
||||
# when no tsdb "last datum" is provided, we just load
|
||||
# some near-term history.
|
||||
# periods = {
|
||||
# 1: {'days': 1},
|
||||
# 60: {'days': 14},
|
||||
# }
|
||||
|
||||
# do a decently sized backfill and load it into storage.
|
||||
periods = {
|
||||
1: {'days': 6},
|
||||
60: {'years': 6},
|
||||
}
|
||||
period_duration: int = periods[timeframe]
|
||||
|
||||
update_start_on_prepend = True
|
||||
|
||||
# NOTE: manually set the "latest" datetime which we intend to
|
||||
# backfill history "until" so as to adhere to the history
|
||||
# settings above when the tsdb is detected as being empty.
|
||||
backfill_until_dt = backfill_from_dt.subtract(**period_duration)
|
||||
|
||||
|
||||
# TODO: can we drop this? without conc i don't think this
|
||||
# is necessary any more?
|
||||
# configure async query throttling
|
||||
# rate = config.get('rate', 1)
|
||||
# XXX: legacy from ``trimeter`` code but unsupported now.
|
||||
# erlangs = config.get('erlangs', 1)
|
||||
# avoid duplicate history frames with a set of datetime frame
|
||||
# starts and associated counts of how many duplicates we see
|
||||
# per time stamp.
|
||||
# starts: Counter[datetime] = Counter()
|
||||
|
||||
# conduct "backward history gap filling" where we push to
|
||||
# the shm buffer until we have history back until the
|
||||
# latest entry loaded from the tsdb's table B)
|
||||
last_start_dt: datetime = backfill_from_dt
|
||||
next_prepend_index: int = backfill_from_shm_index
|
||||
|
||||
while last_start_dt > backfill_until_dt:
|
||||
|
||||
log.debug(
|
||||
f'Requesting {timeframe}s frame ending in {last_start_dt}'
|
||||
)
|
||||
|
||||
try:
|
||||
(
|
||||
array,
|
||||
next_start_dt,
|
||||
next_end_dt,
|
||||
) = await get_hist(
|
||||
timeframe,
|
||||
end_dt=last_start_dt,
|
||||
)
|
||||
|
||||
# broker says there never was or is no more history to pull
|
||||
except DataUnavailable:
|
||||
log.warning(
|
||||
f'NO-MORE-DATA: backend {mod.name} halted history!?'
|
||||
)
|
||||
|
||||
# ugh, what's a better way?
|
||||
# TODO: fwiw, we probably want a way to signal a throttle
|
||||
# condition (eg. with ib) so that we can halt the
|
||||
# request loop until the condition is resolved?
|
||||
return
|
||||
|
||||
# TODO: drop this? see todo above..
|
||||
# if (
|
||||
# next_start_dt in starts
|
||||
# and starts[next_start_dt] <= 6
|
||||
# ):
|
||||
# start_dt = min(starts)
|
||||
# log.warning(
|
||||
# f"{mkt.fqme}: skipping duplicate frame @ {next_start_dt}"
|
||||
# )
|
||||
# starts[start_dt] += 1
|
||||
# await tractor.pause()
|
||||
# continue
|
||||
|
||||
# elif starts[next_start_dt] > 6:
|
||||
# log.warning(
|
||||
# f'NO-MORE-DATA: backend {mod.name} before {next_start_dt}?'
|
||||
# )
|
||||
# return
|
||||
|
||||
# # only update new start point if not-yet-seen
|
||||
# starts[next_start_dt] += 1
|
||||
|
||||
assert array['time'][0] == next_start_dt.timestamp()
|
||||
|
||||
diff = last_start_dt - next_start_dt
|
||||
frame_time_diff_s = diff.seconds
|
||||
|
||||
# frame's worth of sample-period-steps, in seconds
|
||||
frame_size_s = len(array) * timeframe
|
||||
expected_frame_size_s = frame_size_s + timeframe
|
||||
if frame_time_diff_s > expected_frame_size_s:
|
||||
|
||||
# XXX: query result includes a start point prior to our
|
||||
# expected "frame size" and thus is likely some kind of
|
||||
# history gap (eg. market closed period, outage, etc.)
|
||||
# so just report it to console for now.
|
||||
log.warning(
|
||||
f'History frame ending @ {last_start_dt} appears to have a gap:\n'
|
||||
f'{diff} ~= {frame_time_diff_s} seconds'
|
||||
)
|
||||
|
||||
to_push = diff_history(
|
||||
array,
|
||||
prepend_until_dt=backfill_until_dt,
|
||||
)
|
||||
ln = len(to_push)
|
||||
if ln:
|
||||
log.info(f'{ln} bars for {next_start_dt} -> {last_start_dt}')
|
||||
|
||||
else:
|
||||
log.warning(
|
||||
'0 BARS TO PUSH after diff!?\n'
|
||||
f'{next_start_dt} -> {last_start_dt}'
|
||||
)
|
||||
|
||||
# bail gracefully on shm allocation overrun/full
|
||||
# condition
|
||||
try:
|
||||
await shm_push_in_between(
|
||||
shm,
|
||||
to_push,
|
||||
prepend_index=next_prepend_index,
|
||||
update_start_on_prepend=update_start_on_prepend,
|
||||
)
|
||||
await sampler_stream.send({
|
||||
'broadcast_all': {
|
||||
'backfilling': (mkt.fqme, timeframe),
|
||||
},
|
||||
})
|
||||
|
||||
# decrement next prepend point
|
||||
next_prepend_index = next_prepend_index - ln
|
||||
last_start_dt = next_start_dt
|
||||
|
||||
except ValueError as ve:
|
||||
_ve = ve
|
||||
log.error(
|
||||
f'Shm prepend OVERRUN on: {next_start_dt} -> {last_start_dt}?'
|
||||
)
|
||||
|
||||
if next_prepend_index < ln:
|
||||
log.warning(
|
||||
f'Shm buffer can only hold {next_prepend_index} more rows..\n'
|
||||
f'Appending those from recent {ln}-sized frame, no more!'
|
||||
)
|
||||
|
||||
to_push = to_push[-next_prepend_index + 1:]
|
||||
await shm_push_in_between(
|
||||
shm,
|
||||
to_push,
|
||||
prepend_index=next_prepend_index,
|
||||
update_start_on_prepend=update_start_on_prepend,
|
||||
)
|
||||
await sampler_stream.send({
|
||||
'broadcast_all': {
|
||||
'backfilling': (mkt.fqme, timeframe),
|
||||
},
|
||||
})
|
||||
|
||||
# can't push the entire frame? so
|
||||
# push only the amount that can fit..
|
||||
break
|
||||
|
||||
log.info(
|
||||
f'Shm pushed {ln} frame:\n'
|
||||
f'{next_start_dt} -> {last_start_dt}'
|
||||
)
|
||||
|
||||
# FINALLY, maybe write immediately to the tsdb backend for
|
||||
# long-term storage.
|
||||
if (
|
||||
storage is not None
|
||||
and write_tsdb
|
||||
):
|
||||
log.info(
|
||||
f'Writing {ln} frame to storage:\n'
|
||||
f'{next_start_dt} -> {last_start_dt}'
|
||||
)
|
||||
|
||||
# always drop the src asset token for
|
||||
# non-currency-pair like market types (for now)
|
||||
if mkt.dst.atype not in {
|
||||
'crypto',
|
||||
'crypto_currency',
|
||||
'fiat', # a "forex pair"
|
||||
}:
|
||||
# for now, our table key schema is not including
|
||||
# the dst[/src] source asset token.
|
||||
col_sym_key: str = mkt.get_fqme(
|
||||
delim_char='',
|
||||
without_src=True,
|
||||
)
|
||||
else:
|
||||
col_sym_key: str = mkt.get_fqme(delim_char='')
|
||||
|
||||
# TODO: implement parquet append!?
|
||||
await storage.write_ohlcv(
|
||||
col_sym_key,
|
||||
shm.array,
|
||||
timeframe,
|
||||
)
|
||||
else:
|
||||
# finally filled gap
|
||||
log.info(
|
||||
f'Finished filling gap to tsdb start @ {backfill_until_dt}!'
|
||||
)
|
||||
# conduct tsdb timestamp gap detection and backfill any
|
||||
# seemingly missing sequence segments..
|
||||
# TODO: ideally these never exist but somehow it seems
|
||||
# sometimes we're writing zero-ed segments on certain
|
||||
# (teardown) cases?
|
||||
from ._timeseries import detect_null_time_gap
|
||||
|
||||
gap_indices: tuple | None = detect_null_time_gap(shm)
|
||||
while gap_indices:
|
||||
(
|
||||
istart,
|
||||
start,
|
||||
end,
|
||||
iend,
|
||||
) = gap_indices
|
||||
|
||||
start_dt = from_timestamp(start)
|
||||
end_dt = from_timestamp(end)
|
||||
(
|
||||
array,
|
||||
next_start_dt,
|
||||
next_end_dt,
|
||||
) = await get_hist(
|
||||
timeframe,
|
||||
start_dt=start_dt,
|
||||
end_dt=end_dt,
|
||||
)
|
||||
|
||||
# XXX TODO: pretty sure if i plot tsla, btcusdt.binance
|
||||
# and mnq.cme.ib this causes a Qt crash XXDDD
|
||||
|
||||
# make sure we don't overrun the buffer start
|
||||
len_to_push: int = min(iend, array.size)
|
||||
to_push: np.ndarray = array[-len_to_push:]
|
||||
await shm_push_in_between(
|
||||
shm,
|
||||
to_push,
|
||||
prepend_index=iend,
|
||||
update_start_on_prepend=False,
|
||||
)
|
||||
|
||||
# TODO: UI side needs IPC event to update..
|
||||
# - make sure the UI actually always handles
|
||||
# this update!
|
||||
# - remember that in the display side, only refersh this
|
||||
# if the respective history is actually "in view".
|
||||
# loop
|
||||
await sampler_stream.send({
|
||||
'broadcast_all': {
|
||||
'backfilling': (mkt.fqme, timeframe),
|
||||
},
|
||||
})
|
||||
gap_indices: tuple | None = detect_null_time_gap(shm)
|
||||
|
||||
# XXX: extremely important, there can be no checkpoints
|
||||
# in the block above to avoid entering new ``frames``
|
||||
# values while we're pipelining the current ones to
|
||||
# memory...
|
||||
# await sampler_stream.send('broadcast_all')
|
||||
|
||||
# short-circuit (for now)
|
||||
bf_done.set()
|
||||
|
||||
|
||||
async def back_load_from_tsdb(
|
||||
storemod: ModuleType,
|
||||
storage: StorageClient,
|
||||
|
||||
fqme: str,
|
||||
|
||||
tsdb_history: np.ndarray,
|
||||
|
||||
last_tsdb_dt: datetime,
|
||||
latest_start_dt: datetime,
|
||||
latest_end_dt: datetime,
|
||||
|
||||
bf_done: trio.Event,
|
||||
|
||||
timeframe: int,
|
||||
shm: ShmArray,
|
||||
):
|
||||
assert len(tsdb_history)
|
||||
|
||||
# sync to backend history task's query/load completion
|
||||
# if bf_done:
|
||||
# await bf_done.wait()
|
||||
|
||||
# TODO: eventually it'd be nice to not require a shm array/buffer
|
||||
# to accomplish this.. maybe we can do some kind of tsdb direct to
|
||||
# graphics format eventually in a child-actor?
|
||||
if storemod.name == 'nativedb':
|
||||
return
|
||||
|
||||
await tractor.pause()
|
||||
assert shm._first.value == 0
|
||||
|
||||
array = shm.array
|
||||
|
||||
# if timeframe == 1:
|
||||
# times = shm.array['time']
|
||||
# assert (times[1] - times[0]) == 1
|
||||
|
||||
if len(array):
|
||||
shm_last_dt = from_timestamp(
|
||||
shm.array[0]['time']
|
||||
)
|
||||
else:
|
||||
shm_last_dt = None
|
||||
|
||||
if last_tsdb_dt:
|
||||
assert shm_last_dt >= last_tsdb_dt
|
||||
|
||||
# do diff against start index of last frame of history and only
|
||||
# fill in an amount of datums from tsdb allows for most recent
|
||||
# to be loaded into mem *before* tsdb data.
|
||||
if (
|
||||
last_tsdb_dt
|
||||
and latest_start_dt
|
||||
):
|
||||
backfilled_size_s = (
|
||||
latest_start_dt - last_tsdb_dt
|
||||
).seconds
|
||||
# if the shm buffer len is not large enough to contain
|
||||
# all missing data between the most recent backend-queried frame
|
||||
# and the most recent dt-index in the db we warn that we only
|
||||
# want to load a portion of the next tsdb query to fill that
|
||||
# space.
|
||||
log.info(
|
||||
f'{backfilled_size_s} seconds worth of {timeframe}s loaded'
|
||||
)
|
||||
|
||||
# Load TSDB history into shm buffer (for display) if there is
|
||||
# remaining buffer space.
|
||||
|
||||
time_key: str = 'time'
|
||||
if getattr(storemod, 'ohlc_key_map', False):
|
||||
keymap: bidict = storemod.ohlc_key_map
|
||||
time_key: str = keymap.inverse['time']
|
||||
|
||||
# if (
|
||||
# not len(tsdb_history)
|
||||
# ):
|
||||
# return
|
||||
|
||||
tsdb_last_frame_start: datetime = last_tsdb_dt
|
||||
# load as much from storage into shm possible (depends on
|
||||
# user's shm size settings).
|
||||
while shm._first.value > 0:
|
||||
|
||||
tsdb_history = await storage.read_ohlcv(
|
||||
fqme,
|
||||
timeframe=timeframe,
|
||||
end=tsdb_last_frame_start,
|
||||
)
|
||||
|
||||
# # empty query
|
||||
# if not len(tsdb_history):
|
||||
# break
|
||||
|
||||
next_start = tsdb_history[time_key][0]
|
||||
if next_start >= tsdb_last_frame_start:
|
||||
# no earlier data detected
|
||||
break
|
||||
|
||||
else:
|
||||
tsdb_last_frame_start = next_start
|
||||
|
||||
# TODO: see if there's faster multi-field reads:
|
||||
# https://numpy.org/doc/stable/user/basics.rec.html#accessing-multiple-fields
|
||||
# re-index with a `time` and index field
|
||||
prepend_start = shm._first.value
|
||||
|
||||
to_push = tsdb_history[-prepend_start:]
|
||||
shm.push(
|
||||
to_push,
|
||||
|
||||
# insert the history pre a "days worth" of samples
|
||||
# to leave some real-time buffer space at the end.
|
||||
prepend=True,
|
||||
# update_first=False,
|
||||
# start=prepend_start,
|
||||
field_map=storemod.ohlc_key_map,
|
||||
)
|
||||
|
||||
log.info(f'Loaded {to_push.shape} datums from storage')
|
||||
tsdb_last_frame_start = tsdb_history[time_key][0]
|
||||
|
||||
# manually trigger step update to update charts/fsps
|
||||
# which need an incremental update.
|
||||
# NOTE: the way this works is super duper
|
||||
# un-intuitive right now:
|
||||
# - the broadcaster fires a msg to the fsp subsystem.
|
||||
# - fsp subsys then checks for a sample step diff and
|
||||
# possibly recomputes prepended history.
|
||||
# - the fsp then sends back to the parent actor
|
||||
# (usually a chart showing graphics for said fsp)
|
||||
# which tells the chart to conduct a manual full
|
||||
# graphics loop cycle.
|
||||
# await sampler_stream.send('broadcast_all')
|
||||
|
||||
|
||||
async def tsdb_backfill(
|
||||
mod: ModuleType,
|
||||
storemod: ModuleType,
|
||||
tn: trio.Nursery,
|
||||
|
||||
storage: StorageClient,
|
||||
mkt: MktPair,
|
||||
shm: ShmArray,
|
||||
timeframe: float,
|
||||
|
||||
sampler_stream: tractor.MsgStream,
|
||||
|
||||
task_status: TaskStatus[
|
||||
tuple[ShmArray, ShmArray]
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
|
||||
get_hist: Callable[
|
||||
[int, datetime, datetime],
|
||||
tuple[np.ndarray, str]
|
||||
]
|
||||
config: dict[str, int]
|
||||
async with mod.open_history_client(
|
||||
mkt,
|
||||
) as (get_hist, config):
|
||||
log.info(f'{mod} history client returned backfill config: {config}')
|
||||
|
||||
# get latest query's worth of history all the way
|
||||
# back to what is recorded in the tsdb
|
||||
try:
|
||||
array, mr_start_dt, mr_end_dt = await get_hist(
|
||||
timeframe,
|
||||
end_dt=None,
|
||||
)
|
||||
|
||||
# XXX: timeframe not supported for backend (since
|
||||
# above exception type), terminate immediately since
|
||||
# there's no backfilling possible.
|
||||
except DataUnavailable:
|
||||
task_status.started()
|
||||
return
|
||||
|
||||
# TODO: fill in non-zero epoch time values ALWAYS!
|
||||
# hist_shm._array['time'] = np.arange(
|
||||
# start=
|
||||
|
||||
# NOTE: removed for now since it'll always break
|
||||
# on the first 60s of the venue open..
|
||||
# times: np.ndarray = array['time']
|
||||
# # sample period step size in seconds
|
||||
# step_size_s = (
|
||||
# from_timestamp(times[-1])
|
||||
# - from_timestamp(times[-2])
|
||||
# ).seconds
|
||||
|
||||
# if step_size_s not in (1, 60):
|
||||
# log.error(f'Last 2 sample period is off!? -> {step_size_s}')
|
||||
# step_size_s = (
|
||||
# from_timestamp(times[-2])
|
||||
# - from_timestamp(times[-3])
|
||||
# ).seconds
|
||||
|
||||
# NOTE: on the first history, most recent history
|
||||
# frame we PREPEND from the current shm ._last index
|
||||
# and thus a gap between the earliest datum loaded here
|
||||
# and the latest loaded from the tsdb may exist!
|
||||
log.info(f'Pushing {array.size} to shm!')
|
||||
shm.push(
|
||||
array,
|
||||
prepend=True, # append on first frame
|
||||
)
|
||||
backfill_gap_from_shm_index: int = shm._first.value + 1
|
||||
|
||||
# tell parent task to continue
|
||||
task_status.started()
|
||||
|
||||
# loads a (large) frame of data from the tsdb depending
|
||||
# on the db's query size limit; our "nativedb" (using
|
||||
# parquet) generally can load the entire history into mem
|
||||
# but if not then below the remaining history can be lazy
|
||||
# loaded?
|
||||
fqme: str = mkt.fqme
|
||||
tsdb_entry: tuple | None = await storage.load(
|
||||
fqme,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
|
||||
last_tsdb_dt: datetime | None = None
|
||||
if tsdb_entry:
|
||||
(
|
||||
tsdb_history,
|
||||
first_tsdb_dt,
|
||||
last_tsdb_dt,
|
||||
) = tsdb_entry
|
||||
|
||||
# calc the index from which the tsdb data should be
|
||||
# prepended, presuming there is a gap between the
|
||||
# latest frame (loaded/read above) and the latest
|
||||
# sample loaded from the tsdb.
|
||||
backfill_diff: Duration = mr_start_dt - last_tsdb_dt
|
||||
offset_s: float = backfill_diff.in_seconds()
|
||||
offset_samples: int = round(offset_s / timeframe)
|
||||
|
||||
# TODO: see if there's faster multi-field reads:
|
||||
# https://numpy.org/doc/stable/user/basics.rec.html#accessing-multiple-fields
|
||||
# re-index with a `time` and index field
|
||||
prepend_start = shm._first.value - offset_samples + 1
|
||||
|
||||
# tsdb history is so far in the past we can't fit it in
|
||||
# shm buffer space so simply don't load it!
|
||||
if prepend_start > 0:
|
||||
to_push = tsdb_history[-prepend_start:]
|
||||
shm.push(
|
||||
to_push,
|
||||
|
||||
# insert the history pre a "days worth" of samples
|
||||
# to leave some real-time buffer space at the end.
|
||||
prepend=True,
|
||||
# update_first=False,
|
||||
start=prepend_start,
|
||||
field_map=storemod.ohlc_key_map,
|
||||
)
|
||||
|
||||
log.info(f'Loaded {to_push.shape} datums from storage')
|
||||
|
||||
# TODO: maybe start history anal and load missing "history
|
||||
# gaps" via backend..
|
||||
|
||||
if timeframe not in (1, 60):
|
||||
raise ValueError(
|
||||
'`piker` only needs to support 1m and 1s sampling '
|
||||
'but ur api is trying to deliver a longer '
|
||||
f'timeframe of {timeframe} seconds..\n'
|
||||
'So yuh.. dun do dat brudder.'
|
||||
)
|
||||
# if there is a gap to backfill from the first
|
||||
# history frame until the last datum loaded from the tsdb
|
||||
# continue that now in the background
|
||||
bf_done = await tn.start(
|
||||
partial(
|
||||
start_backfill,
|
||||
get_hist,
|
||||
mod,
|
||||
mkt,
|
||||
shm,
|
||||
timeframe,
|
||||
|
||||
backfill_from_shm_index=backfill_gap_from_shm_index,
|
||||
backfill_from_dt=mr_start_dt,
|
||||
|
||||
sampler_stream=sampler_stream,
|
||||
|
||||
backfill_until_dt=last_tsdb_dt,
|
||||
storage=storage,
|
||||
)
|
||||
)
|
||||
|
||||
# if len(hist_shm.array) < 2:
|
||||
# TODO: there's an edge case here to solve where if the last
|
||||
# frame before market close (at least on ib) was pushed and
|
||||
# there was only "1 new" row pushed from the first backfill
|
||||
# query-iteration, then the sample step sizing calcs will
|
||||
# break upstream from here since you can't diff on at least
|
||||
# 2 steps... probably should also add logic to compute from
|
||||
# the tsdb series and stash that somewhere as meta data on
|
||||
# the shm buffer?.. no se.
|
||||
|
||||
# backload any further data from tsdb (concurrently per
|
||||
# timeframe) if not all data was able to be loaded (in memory)
|
||||
# from the ``StorageClient.load()`` call above.
|
||||
try:
|
||||
await trio.sleep_forever()
|
||||
finally:
|
||||
return
|
||||
|
||||
# IF we need to continue backloading incrementally from the
|
||||
# tsdb client..
|
||||
tn.start_soon(
|
||||
back_load_from_tsdb,
|
||||
|
||||
storemod,
|
||||
storage,
|
||||
fqme,
|
||||
|
||||
tsdb_history,
|
||||
last_tsdb_dt,
|
||||
mr_start_dt,
|
||||
mr_end_dt,
|
||||
bf_done,
|
||||
|
||||
timeframe,
|
||||
shm,
|
||||
)
|
||||
|
||||
|
||||
async def manage_history(
|
||||
mod: ModuleType,
|
||||
bus: _FeedsBus,
|
||||
mkt: MktPair,
|
||||
some_data_ready: trio.Event,
|
||||
feed_is_live: trio.Event,
|
||||
timeframe: float = 60, # in seconds
|
||||
|
||||
task_status: TaskStatus[
|
||||
tuple[ShmArray, ShmArray]
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Load and manage historical data including the loading of any
|
||||
available series from any connected tsdb as well as conduct
|
||||
real-time update of both that existing db and the allocated
|
||||
shared memory buffer.
|
||||
|
||||
Init sequence:
|
||||
- allocate shm (numpy array) buffers for 60s & 1s sample rates
|
||||
- configure "zero index" for each buffer: the index where
|
||||
history will prepended *to* and new live data will be
|
||||
appened *from*.
|
||||
- open a ``.storage.StorageClient`` and load any existing tsdb
|
||||
history as well as (async) start a backfill task which loads
|
||||
missing (newer) history from the data provider backend:
|
||||
- tsdb history is loaded first and pushed to shm ASAP.
|
||||
- the backfill task loads the most recent history before
|
||||
unblocking its parent task, so that the `ShmArray._last` is
|
||||
up to date to allow the OHLC sampler to begin writing new
|
||||
samples as the correct buffer index once the provider feed
|
||||
engages.
|
||||
|
||||
'''
|
||||
# TODO: is there a way to make each shm file key
|
||||
# actor-tree-discovery-addr unique so we avoid collisions
|
||||
# when doing tests which also allocate shms for certain instruments
|
||||
# that may be in use on the system by some other running daemons?
|
||||
# from tractor._state import _runtime_vars
|
||||
# port = _runtime_vars['_root_mailbox'][1]
|
||||
|
||||
uid: tuple = tractor.current_actor().uid
|
||||
name, uuid = uid
|
||||
service: str = name.rstrip(f'.{mod.name}')
|
||||
fqme: str = mkt.get_fqme(delim_char='')
|
||||
|
||||
# (maybe) allocate shm array for this broker/symbol which will
|
||||
# be used for fast near-term history capture and processing.
|
||||
hist_shm, opened = maybe_open_shm_array(
|
||||
size=_default_hist_size,
|
||||
append_start_index=_hist_buffer_start,
|
||||
|
||||
key=f'piker.{service}[{uuid[:16]}].{fqme}.hist',
|
||||
|
||||
# use any broker defined ohlc dtype:
|
||||
dtype=getattr(mod, '_ohlc_dtype', def_iohlcv_fields),
|
||||
|
||||
# we expect the sub-actor to write
|
||||
readonly=False,
|
||||
)
|
||||
hist_zero_index = hist_shm.index - 1
|
||||
|
||||
# TODO: history validation
|
||||
if not opened:
|
||||
raise RuntimeError(
|
||||
"Persistent shm for sym was already open?!"
|
||||
)
|
||||
|
||||
rt_shm, opened = maybe_open_shm_array(
|
||||
size=_default_rt_size,
|
||||
append_start_index=_rt_buffer_start,
|
||||
key=f'piker.{service}[{uuid[:16]}].{fqme}.rt',
|
||||
|
||||
# use any broker defined ohlc dtype:
|
||||
dtype=getattr(mod, '_ohlc_dtype', def_iohlcv_fields),
|
||||
|
||||
# we expect the sub-actor to write
|
||||
readonly=False,
|
||||
)
|
||||
|
||||
# (for now) set the rt (hft) shm array with space to prepend
|
||||
# only a few days worth of 1s history.
|
||||
days: int = 2
|
||||
start_index: int = days*_secs_in_day
|
||||
rt_shm._first.value = start_index
|
||||
rt_shm._last.value = start_index
|
||||
rt_zero_index = rt_shm.index - 1
|
||||
|
||||
if not opened:
|
||||
raise RuntimeError(
|
||||
"Persistent shm for sym was already open?!"
|
||||
)
|
||||
|
||||
open_history_client = getattr(
|
||||
mod,
|
||||
'open_history_client',
|
||||
)
|
||||
assert open_history_client
|
||||
|
||||
# TODO: maybe it should be a subpkg of `.data`?
|
||||
from piker import storage
|
||||
|
||||
async with (
|
||||
storage.open_storage_client() as (storemod, client),
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
log.info(
|
||||
f'Connecting to storage backend `{storemod.name}`:\n'
|
||||
f'location: {client.address}\n'
|
||||
f'db cardinality: {client.cardinality}\n'
|
||||
# TODO: show backend config, eg:
|
||||
# - network settings
|
||||
# - storage size with compression
|
||||
# - number of loaded time series?
|
||||
)
|
||||
|
||||
# NOTE: this call ONLY UNBLOCKS once the latest-most frame
|
||||
# (i.e. history just before the live feed latest datum) of
|
||||
# history has been loaded and written to the shm buffer:
|
||||
# - the backfiller task can write in reverse chronological
|
||||
# to the shm and tsdb
|
||||
# - the tsdb data can be loaded immediately and the
|
||||
# backfiller can do a single append from it's end datum and
|
||||
# then prepends backward to that from the current time
|
||||
# step.
|
||||
tf2mem: dict = {
|
||||
1: rt_shm,
|
||||
60: hist_shm,
|
||||
}
|
||||
async with open_sample_stream(
|
||||
period_s=1.,
|
||||
shms_by_period={
|
||||
1.: rt_shm.token,
|
||||
60.: hist_shm.token,
|
||||
},
|
||||
|
||||
# NOTE: we want to only open a stream for doing
|
||||
# broadcasts on backfill operations, not receive the
|
||||
# sample index-stream (since there's no code in this
|
||||
# data feed layer that needs to consume it).
|
||||
open_index_stream=True,
|
||||
sub_for_broadcasts=False,
|
||||
|
||||
) as sample_stream:
|
||||
# register 1s and 1m buffers with the global incrementer task
|
||||
log.info(f'Connected to sampler stream: {sample_stream}')
|
||||
|
||||
for timeframe in [60, 1]:
|
||||
await tn.start(
|
||||
tsdb_backfill,
|
||||
mod,
|
||||
storemod,
|
||||
tn,
|
||||
# bus,
|
||||
client,
|
||||
mkt,
|
||||
tf2mem[timeframe],
|
||||
timeframe,
|
||||
|
||||
sample_stream,
|
||||
)
|
||||
|
||||
# indicate to caller that feed can be delivered to
|
||||
# remote requesting client since we've loaded history
|
||||
# data that can be used.
|
||||
some_data_ready.set()
|
||||
|
||||
# wait for a live feed before starting the sampler.
|
||||
await feed_is_live.wait()
|
||||
|
||||
# yield back after client connect with filled shm
|
||||
task_status.started((
|
||||
hist_zero_index,
|
||||
hist_shm,
|
||||
rt_zero_index,
|
||||
rt_shm,
|
||||
))
|
||||
|
||||
# history retreival loop depending on user interaction
|
||||
# and thus a small RPC-prot for remotely controllinlg
|
||||
# what data is loaded for viewing.
|
||||
await trio.sleep_forever()
|
||||
|
|
@ -113,9 +113,9 @@ def validate_backend(
|
|||
)
|
||||
if ep is None:
|
||||
log.warning(
|
||||
f'Provider backend {mod.name!r} is missing '
|
||||
f'{daemon_name!r} support?\n'
|
||||
f'|_module endpoint-func missing: {name!r}\n'
|
||||
f'Provider backend {mod.name} is missing '
|
||||
f'{daemon_name} support :(\n'
|
||||
f'The following endpoint is missing: {name}'
|
||||
)
|
||||
|
||||
inits: list[
|
||||
|
|
|
|||
|
|
@ -26,10 +26,7 @@ from ._api import (
|
|||
maybe_mk_fsp_shm,
|
||||
Fsp,
|
||||
)
|
||||
from ._engine import (
|
||||
cascade,
|
||||
Cascade,
|
||||
)
|
||||
from ._engine import cascade
|
||||
from ._volume import (
|
||||
dolla_vlm,
|
||||
flow_rates,
|
||||
|
|
@ -38,7 +35,6 @@ from ._volume import (
|
|||
|
||||
__all__: list[str] = [
|
||||
'cascade',
|
||||
'Cascade',
|
||||
'maybe_mk_fsp_shm',
|
||||
'Fsp',
|
||||
'dolla_vlm',
|
||||
|
|
@ -50,12 +46,9 @@ __all__: list[str] = [
|
|||
async def latency(
|
||||
source: 'TickStream[Dict[str, float]]', # noqa
|
||||
ohlcv: np.ndarray
|
||||
|
||||
) -> AsyncIterator[np.ndarray]:
|
||||
'''
|
||||
Latency measurements, broker to piker.
|
||||
|
||||
'''
|
||||
"""Latency measurements, broker to piker.
|
||||
"""
|
||||
# TODO: do we want to offer yielding this async
|
||||
# before the rt data connection comes up?
|
||||
|
||||
|
|
|
|||
|
|
@ -200,13 +200,9 @@ def maybe_mk_fsp_shm(
|
|||
)
|
||||
|
||||
# (attempt to) uniquely key the fsp shm buffers
|
||||
# Use hash for macOS compatibility (31 char limit)
|
||||
import hashlib
|
||||
actor_name, uuid = tractor.current_actor().uid
|
||||
# Create short hash of sym and target name
|
||||
content = f'{sym}.{target.name}'
|
||||
content_hash = hashlib.md5(content.encode()).hexdigest()[:8]
|
||||
key: str = f'{uuid[:8]}_{content_hash}.fsp'
|
||||
uuid_snip: str = uuid[:16]
|
||||
key: str = f'piker.{actor_name}[{uuid_snip}].{sym}.{target.name}'
|
||||
|
||||
shm, opened = maybe_open_shm_array(
|
||||
key,
|
||||
|
|
|
|||
|
|
@ -18,13 +18,13 @@
|
|||
core task logic for processing chains
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
from contextlib import asynccontextmanager as acm
|
||||
from dataclasses import dataclass
|
||||
from functools import partial
|
||||
from typing import (
|
||||
AsyncIterator,
|
||||
Callable,
|
||||
TYPE_CHECKING,
|
||||
Optional,
|
||||
Union,
|
||||
)
|
||||
|
||||
import numpy as np
|
||||
|
|
@ -33,13 +33,13 @@ from trio_typing import TaskStatus
|
|||
import tractor
|
||||
from tractor.msg import NamespacePath
|
||||
|
||||
from piker.types import Struct
|
||||
from ..log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
from ..log import get_logger, get_console_log
|
||||
from .. import data
|
||||
from ..data.flows import Flume
|
||||
from ..data import attach_shm_array
|
||||
from ..data.feed import (
|
||||
Flume,
|
||||
Feed,
|
||||
)
|
||||
from ..data._sharedmem import ShmArray
|
||||
from ..data._sampling import (
|
||||
_default_delay_s,
|
||||
|
|
@ -53,12 +53,15 @@ from ._api import (
|
|||
)
|
||||
from ..toolz import Profiler
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..data.feed import Feed
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskTracker:
|
||||
complete: trio.Event
|
||||
cs: trio.CancelScope
|
||||
|
||||
|
||||
async def filter_quotes_by_sym(
|
||||
|
||||
sym: str,
|
||||
|
|
@ -79,170 +82,30 @@ async def filter_quotes_by_sym(
|
|||
if quote:
|
||||
yield quote
|
||||
|
||||
# TODO: unifying the abstractions in this FSP subsys/layer:
|
||||
# -[ ] move the `.data.flows.Flume` type into this
|
||||
# module/subsys/pkg?
|
||||
# -[ ] ideas for further abstractions as per
|
||||
# - https://github.com/pikers/piker/issues/216,
|
||||
# - https://github.com/pikers/piker/issues/270:
|
||||
# - a (financial signal) ``Flow`` would be the a "collection" of such
|
||||
# minmial cascades. Some engineering based jargon concepts:
|
||||
# - https://en.wikipedia.org/wiki/Signal_chain
|
||||
# - https://en.wikipedia.org/wiki/Daisy_chain_(electrical_engineering)
|
||||
# - https://en.wikipedia.org/wiki/Audio_signal_flow
|
||||
# - https://en.wikipedia.org/wiki/Digital_signal_processing#Implementation
|
||||
# - https://en.wikipedia.org/wiki/Dataflow_programming
|
||||
# - https://en.wikipedia.org/wiki/Signal_programming
|
||||
# - https://en.wikipedia.org/wiki/Incremental_computing
|
||||
# - https://en.wikipedia.org/wiki/Signal-flow_graph
|
||||
# - https://en.wikipedia.org/wiki/Signal-flow_graph#Basic_components
|
||||
|
||||
# -[ ] we probably want to eval THE BELOW design and unify with the
|
||||
# proto `TaskManager` in the `tractor` dev branch as well as with
|
||||
# our below idea for `Cascade`:
|
||||
# - https://github.com/goodboy/tractor/pull/363
|
||||
class Cascade(Struct):
|
||||
'''
|
||||
As per sig-proc engineering parlance, this is a chaining of
|
||||
`Flume`s, which are themselves collections of "Streams"
|
||||
implemented currently via `ShmArray`s.
|
||||
async def fsp_compute(
|
||||
|
||||
A `Cascade` is be the minimal "connection" of 2 `Flumes`
|
||||
as per circuit parlance:
|
||||
https://en.wikipedia.org/wiki/Two-port_network#Cascade_connection
|
||||
|
||||
TODO:
|
||||
-[ ] could cover the combination of our `FspAdmin` and the
|
||||
backend `.fsp._engine` related machinery to "connect" one flume
|
||||
to another?
|
||||
|
||||
'''
|
||||
# TODO: make these `Flume`s
|
||||
src: Flume
|
||||
dst: Flume
|
||||
tn: trio.Nursery
|
||||
fsp: Fsp # UI-side middleware ctl API
|
||||
|
||||
# filled during cascade/.bind_func() (fsp_compute) init phases
|
||||
bind_func: Callable | None = None
|
||||
complete: trio.Event | None = None
|
||||
cs: trio.CancelScope | None = None
|
||||
client_stream: tractor.MsgStream | None = None
|
||||
|
||||
async def resync(self) -> int:
|
||||
# TODO: adopt an incremental update engine/approach
|
||||
# where possible here eventually!
|
||||
log.info(f're-syncing fsp {self.fsp.name} to source')
|
||||
self.cs.cancel()
|
||||
await self.complete.wait()
|
||||
index: int = await self.tn.start(self.bind_func)
|
||||
|
||||
# always trigger UI refresh after history update,
|
||||
# see ``piker.ui._fsp.FspAdmin.open_chain()`` and
|
||||
# ``piker.ui._display.trigger_update()``.
|
||||
dst_shm: ShmArray = self.dst.rt_shm
|
||||
await self.client_stream.send({
|
||||
'fsp_update': {
|
||||
'key': dst_shm.token,
|
||||
'first': dst_shm._first.value,
|
||||
'last': dst_shm._last.value,
|
||||
}
|
||||
})
|
||||
return index
|
||||
|
||||
def is_synced(self) -> tuple[bool, int, int]:
|
||||
'''
|
||||
Predicate to dertmine if a destination FSP
|
||||
output array is aligned to its source array.
|
||||
|
||||
'''
|
||||
src_shm: ShmArray = self.src.rt_shm
|
||||
dst_shm: ShmArray = self.dst.rt_shm
|
||||
step_diff = src_shm.index - dst_shm.index
|
||||
len_diff = abs(len(src_shm.array) - len(dst_shm.array))
|
||||
synced: bool = not (
|
||||
# the source is likely backfilling and we must
|
||||
# sync history calculations
|
||||
len_diff > 2
|
||||
|
||||
# we aren't step synced to the source and may be
|
||||
# leading/lagging by a step
|
||||
or step_diff > 1
|
||||
or step_diff < 0
|
||||
)
|
||||
if not synced:
|
||||
fsp: Fsp = self.fsp
|
||||
log.warning(
|
||||
f'***DESYNCED fsp***\n'
|
||||
f'------------------\n'
|
||||
f'ns-path: {fsp.ns_path!r}\n'
|
||||
f'shm-token: {src_shm.token}\n'
|
||||
f'step_diff: {step_diff}\n'
|
||||
f'len_diff: {len_diff}\n'
|
||||
)
|
||||
return (
|
||||
synced,
|
||||
step_diff,
|
||||
len_diff,
|
||||
)
|
||||
|
||||
async def poll_and_sync_to_step(self) -> int:
|
||||
synced, step_diff, _ = self.is_synced()
|
||||
while not synced:
|
||||
await self.resync()
|
||||
synced, step_diff, _ = self.is_synced()
|
||||
|
||||
return step_diff
|
||||
|
||||
@acm
|
||||
async def open_edge(
|
||||
self,
|
||||
bind_func: Callable,
|
||||
) -> int:
|
||||
self.bind_func = bind_func
|
||||
index = await self.tn.start(bind_func)
|
||||
yield index
|
||||
# TODO: what do we want on teardown/error?
|
||||
# -[ ] dynamic reconnection after update?
|
||||
|
||||
|
||||
async def connect_streams(
|
||||
casc: Cascade,
|
||||
mkt: MktPair,
|
||||
flume: Flume,
|
||||
quote_stream: trio.abc.ReceiveChannel,
|
||||
src: Flume,
|
||||
dst: Flume,
|
||||
|
||||
edge_func: Callable,
|
||||
src: ShmArray,
|
||||
dst: ShmArray,
|
||||
|
||||
func: Callable,
|
||||
|
||||
# attach_stream: bool = False,
|
||||
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Stream and per-sample compute and write the cascade of
|
||||
2 `Flumes`/streams given some operating `func`.
|
||||
|
||||
https://en.wikipedia.org/wiki/Signal-flow_graph#Basic_components
|
||||
|
||||
Not literally, but something like:
|
||||
|
||||
edge_func(Flume_in) -> Flume_out
|
||||
|
||||
'''
|
||||
profiler = Profiler(
|
||||
delayed=False,
|
||||
disabled=True
|
||||
)
|
||||
|
||||
# TODO: just pull it from src.mkt.fqme no?
|
||||
# fqme: str = mkt.fqme
|
||||
fqme: str = src.mkt.fqme
|
||||
|
||||
# TODO: dynamic introspection of what the underlying (vertex)
|
||||
# function actually requires from input node (flumes) then
|
||||
# deliver those inputs as part of a graph "compilation" step?
|
||||
out_stream = edge_func(
|
||||
fqme = mkt.fqme
|
||||
out_stream = func(
|
||||
|
||||
# TODO: do we even need this if we do the feed api right?
|
||||
# shouldn't a local stream do this before we get a handle
|
||||
|
|
@ -250,21 +113,20 @@ async def connect_streams(
|
|||
# async itertools style?
|
||||
filter_quotes_by_sym(fqme, quote_stream),
|
||||
|
||||
# XXX: currently the ``ohlcv`` arg, but we should allow
|
||||
# (dynamic) requests for src flume (node) streams?
|
||||
src.rt_shm,
|
||||
# XXX: currently the ``ohlcv`` arg
|
||||
flume.rt_shm,
|
||||
)
|
||||
|
||||
# HISTORY COMPUTE PHASE
|
||||
# conduct a single iteration of fsp with historical bars input
|
||||
# and get historical output.
|
||||
history_output: (
|
||||
dict[str, np.ndarray] # multi-output case
|
||||
| np.ndarray, # single output case
|
||||
)
|
||||
history_output: Union[
|
||||
dict[str, np.ndarray], # multi-output case
|
||||
np.ndarray, # single output case
|
||||
]
|
||||
history_output = await anext(out_stream)
|
||||
|
||||
func_name = edge_func.__name__
|
||||
func_name = func.__name__
|
||||
profiler(f'{func_name} generated history')
|
||||
|
||||
# build struct array with an 'index' field to push as history
|
||||
|
|
@ -272,12 +134,10 @@ async def connect_streams(
|
|||
# TODO: push using a[['f0', 'f1', .., 'fn']] = .. syntax no?
|
||||
# if the output array is multi-field then push
|
||||
# each respective field.
|
||||
dst_shm: ShmArray = dst.rt_shm
|
||||
fields = getattr(dst_shm.array.dtype, 'fields', None).copy()
|
||||
fields = getattr(dst.array.dtype, 'fields', None).copy()
|
||||
fields.pop('index')
|
||||
history_by_field: np.ndarray | None = None
|
||||
src_shm: ShmArray = src.rt_shm
|
||||
src_time = src_shm.array['time']
|
||||
history_by_field: Optional[np.ndarray] = None
|
||||
src_time = src.array['time']
|
||||
|
||||
if (
|
||||
fields and
|
||||
|
|
@ -296,7 +156,7 @@ async def connect_streams(
|
|||
if history_by_field is None:
|
||||
|
||||
if output is None:
|
||||
length = len(src_shm.array)
|
||||
length = len(src.array)
|
||||
else:
|
||||
length = len(output)
|
||||
|
||||
|
|
@ -305,7 +165,7 @@ async def connect_streams(
|
|||
# will be pushed to shm.
|
||||
history_by_field = np.zeros(
|
||||
length,
|
||||
dtype=dst_shm.array.dtype
|
||||
dtype=dst.array.dtype
|
||||
)
|
||||
|
||||
if output is None:
|
||||
|
|
@ -322,13 +182,13 @@ async def connect_streams(
|
|||
)
|
||||
history_by_field = np.zeros(
|
||||
len(history_output),
|
||||
dtype=dst_shm.array.dtype
|
||||
dtype=dst.array.dtype
|
||||
)
|
||||
history_by_field[func_name] = history_output
|
||||
|
||||
history_by_field['time'] = src_time[-len(history_by_field):]
|
||||
|
||||
history_output['time'] = src_shm.array['time']
|
||||
history_output['time'] = src.array['time']
|
||||
|
||||
# TODO: XXX:
|
||||
# THERE'S A BIG BUG HERE WITH THE `index` field since we're
|
||||
|
|
@ -341,11 +201,11 @@ async def connect_streams(
|
|||
# is `index` aware such that historical data can be indexed
|
||||
# relative to the true first datum? Not sure if this is sane
|
||||
# for incremental compuations.
|
||||
first = dst_shm._first.value = src_shm._first.value
|
||||
first = dst._first.value = src._first.value
|
||||
|
||||
# TODO: can we use this `start` flag instead of the manual
|
||||
# setting above?
|
||||
index = dst_shm.push(
|
||||
index = dst.push(
|
||||
history_by_field,
|
||||
start=first,
|
||||
)
|
||||
|
|
@ -356,9 +216,12 @@ async def connect_streams(
|
|||
# setup a respawn handle
|
||||
with trio.CancelScope() as cs:
|
||||
|
||||
casc.cs = cs
|
||||
casc.complete = trio.Event()
|
||||
task_status.started(index)
|
||||
# TODO: might be better to just make a "restart" method where
|
||||
# the target task is spawned implicitly and then the event is
|
||||
# set via some higher level api? At that poing we might as well
|
||||
# be writing a one-cancels-one nursery though right?
|
||||
tracker = TaskTracker(trio.Event(), cs)
|
||||
task_status.started((tracker, index))
|
||||
|
||||
profiler(f'{func_name} yield last index')
|
||||
|
||||
|
|
@ -372,12 +235,12 @@ async def connect_streams(
|
|||
log.debug(f"{func_name}: {processed}")
|
||||
key, output = processed
|
||||
# dst.array[-1][key] = output
|
||||
dst_shm.array[[key, 'time']][-1] = (
|
||||
dst.array[[key, 'time']][-1] = (
|
||||
output,
|
||||
# TODO: what about pushing ``time.time_ns()``
|
||||
# in which case we'll need to round at the graphics
|
||||
# processing / sampling layer?
|
||||
src_shm.array[-1]['time']
|
||||
src.array[-1]['time']
|
||||
)
|
||||
|
||||
# NOTE: for now we aren't streaming this to the consumer
|
||||
|
|
@ -389,7 +252,7 @@ async def connect_streams(
|
|||
# N-consumers who subscribe for the real-time output,
|
||||
# which we'll likely want to implement using local-mem
|
||||
# chans for the fan out?
|
||||
# index = src_shm.index
|
||||
# index = src.index
|
||||
# if attach_stream:
|
||||
# await client_stream.send(index)
|
||||
|
||||
|
|
@ -399,25 +262,26 @@ async def connect_streams(
|
|||
# log.info(f'FSP quote too fast: {hz}')
|
||||
# last = time.time()
|
||||
finally:
|
||||
casc.complete.set()
|
||||
tracker.complete.set()
|
||||
|
||||
|
||||
@tractor.context
|
||||
async def cascade(
|
||||
|
||||
ctx: tractor.Context,
|
||||
|
||||
# data feed key
|
||||
fqme: str,
|
||||
|
||||
# flume pair cascaded using an "edge function"
|
||||
src_flume_addr: dict,
|
||||
dst_flume_addr: dict,
|
||||
src_shm_token: dict,
|
||||
dst_shm_token: tuple[str, np.dtype],
|
||||
|
||||
ns_path: NamespacePath,
|
||||
|
||||
shm_registry: dict[str, _Token],
|
||||
|
||||
zero_on_step: bool = False,
|
||||
loglevel: str|None = None,
|
||||
loglevel: Optional[str] = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -431,26 +295,10 @@ async def cascade(
|
|||
)
|
||||
|
||||
if loglevel:
|
||||
log = get_console_log(
|
||||
loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
# XXX TODO!
|
||||
# figure out why this writes a dict to,
|
||||
# `tractor._state._runtime_vars['_root_mailbox']`
|
||||
# XD .. wtf
|
||||
# TODO, solve this as reported in,
|
||||
# https://www.pikers.dev/pikers/piker/issues/70
|
||||
# await tractor.pause()
|
||||
get_console_log(loglevel)
|
||||
|
||||
src: Flume = Flume.from_msg(src_flume_addr)
|
||||
dst: Flume = Flume.from_msg(
|
||||
dst_flume_addr,
|
||||
readonly=False,
|
||||
)
|
||||
|
||||
# src: ShmArray = attach_shm_array(token=src_shm_token)
|
||||
# dst: ShmArray = attach_shm_array(readonly=False, token=dst_shm_token)
|
||||
src = attach_shm_array(token=src_shm_token)
|
||||
dst = attach_shm_array(readonly=False, token=dst_shm_token)
|
||||
|
||||
reg = _load_builtins()
|
||||
lines = '\n'.join([f'{key.rpartition(":")[2]} => {key}' for key in reg])
|
||||
|
|
@ -458,11 +306,11 @@ async def cascade(
|
|||
f'Registered FSP set:\n{lines}'
|
||||
)
|
||||
|
||||
# NOTE XXX: update actorlocal flows table which registers
|
||||
# readonly "instances" of this fsp for symbol/source so that
|
||||
# consumer fsps can look it up by source + fsp.
|
||||
# TODO: ugh i hate this wind/unwind to list over the wire but
|
||||
# not sure how else to do it.
|
||||
# update actorlocal flows table which registers
|
||||
# readonly "instances" of this fsp for symbol/source
|
||||
# so that consumer fsps can look it up by source + fsp.
|
||||
# TODO: ugh i hate this wind/unwind to list over the wire
|
||||
# but not sure how else to do it.
|
||||
for (token, fsp_name, dst_token) in shm_registry:
|
||||
Fsp._flow_registry[(
|
||||
_Token.from_msg(token),
|
||||
|
|
@ -472,20 +320,16 @@ async def cascade(
|
|||
fsp: Fsp = reg.get(
|
||||
NamespacePath(ns_path)
|
||||
)
|
||||
func: Callable = fsp.func
|
||||
func = fsp.func
|
||||
|
||||
if not func:
|
||||
# TODO: assume it's a func target path
|
||||
raise ValueError(f'Unknown fsp target: {ns_path}')
|
||||
|
||||
_fqme: str = src.mkt.fqme
|
||||
assert _fqme == fqme
|
||||
|
||||
# open a data feed stream with requested broker
|
||||
feed: Feed
|
||||
async with data.feed.maybe_open_feed(
|
||||
fqmes=[fqme],
|
||||
loglevel=loglevel,
|
||||
[fqme],
|
||||
|
||||
# TODO throttle tick outputs from *this* daemon since
|
||||
# it'll emit tons of ticks due to the throttle only
|
||||
|
|
@ -495,144 +339,177 @@ async def cascade(
|
|||
|
||||
) as feed:
|
||||
|
||||
flume: Flume = feed.flumes[fqme]
|
||||
# XXX: can't do this since flume.feed will be set XD
|
||||
# assert flume == src
|
||||
assert flume.mkt == src.mkt
|
||||
mkt: MktPair = flume.mkt
|
||||
|
||||
# NOTE: FOR NOW, sanity checks around the feed as being
|
||||
# always the src flume (until we get to fancier/lengthier
|
||||
# chains/graphs.
|
||||
assert src.rt_shm.token == flume.rt_shm.token
|
||||
|
||||
# XXX: won't work bc the _hist_shm_token value will be
|
||||
# list[list] after IPC..
|
||||
# assert flume.to_msg() == src_flume_addr
|
||||
|
||||
flume = feed.flumes[fqme]
|
||||
mkt = flume.mkt
|
||||
assert src.token == flume.rt_shm.token
|
||||
profiler(f'{func}: feed up')
|
||||
|
||||
func_name: str = func.__name__
|
||||
func_name = func.__name__
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(), # avoid multi-taskc tb in console
|
||||
trio.open_nursery() as tn,
|
||||
trio.open_nursery() as n,
|
||||
):
|
||||
# TODO: might be better to just make a "restart" method where
|
||||
# the target task is spawned implicitly and then the event is
|
||||
# set via some higher level api? At that poing we might as well
|
||||
# be writing a one-cancels-one nursery though right?
|
||||
casc = Cascade(
|
||||
src,
|
||||
dst,
|
||||
tn,
|
||||
fsp,
|
||||
)
|
||||
|
||||
# TODO: this seems like it should be wrapped somewhere?
|
||||
fsp_target = partial(
|
||||
connect_streams,
|
||||
casc=casc,
|
||||
|
||||
fsp_compute,
|
||||
mkt=mkt,
|
||||
flume=flume,
|
||||
quote_stream=flume.stream,
|
||||
|
||||
# flumes and shm passthrough
|
||||
# shm
|
||||
src=src,
|
||||
dst=dst,
|
||||
|
||||
# chain function which takes src flume input(s)
|
||||
# and renders dst flume output(s)
|
||||
edge_func=func
|
||||
# target
|
||||
func=func
|
||||
)
|
||||
async with casc.open_edge(
|
||||
bind_func=fsp_target,
|
||||
) as index:
|
||||
# casc.bind_func = fsp_target
|
||||
# index = await tn.start(fsp_target)
|
||||
dst_shm: ShmArray = dst.rt_shm
|
||||
src_shm: ShmArray = src.rt_shm
|
||||
|
||||
if zero_on_step:
|
||||
last = dst.rt_shm.array[-1:]
|
||||
zeroed = np.zeros(last.shape, dtype=last.dtype)
|
||||
tracker, index = await n.start(fsp_target)
|
||||
|
||||
profiler(f'{func_name}: fsp up')
|
||||
if zero_on_step:
|
||||
last = dst.array[-1:]
|
||||
zeroed = np.zeros(last.shape, dtype=last.dtype)
|
||||
|
||||
# sync to client-side actor
|
||||
await ctx.started(index)
|
||||
profiler(f'{func_name}: fsp up')
|
||||
|
||||
# XXX: rt stream with client which we MUST
|
||||
# open here (and keep it open) in order to make
|
||||
# incremental "updates" as history prepends take
|
||||
# place.
|
||||
async with ctx.open_stream() as client_stream:
|
||||
casc.client_stream: tractor.MsgStream = client_stream
|
||||
# sync client
|
||||
await ctx.started(index)
|
||||
|
||||
s, step, ld = casc.is_synced()
|
||||
# XXX: rt stream with client which we MUST
|
||||
# open here (and keep it open) in order to make
|
||||
# incremental "updates" as history prepends take
|
||||
# place.
|
||||
async with ctx.open_stream() as client_stream:
|
||||
|
||||
# detect sample period step for subscription to increment
|
||||
# signal
|
||||
times = src.rt_shm.array['time']
|
||||
if len(times) > 1:
|
||||
last_ts = times[-1]
|
||||
delay_s: float = float(last_ts - times[times != last_ts][-1])
|
||||
else:
|
||||
# our default "HFT" sample rate.
|
||||
delay_s: float = _default_delay_s
|
||||
# TODO: these likely should all become
|
||||
# methods of this ``TaskLifetime`` or wtv
|
||||
# abstraction..
|
||||
async def resync(
|
||||
tracker: TaskTracker,
|
||||
|
||||
# sub and increment the underlying shared memory buffer
|
||||
# on every step msg received from the global `samplerd`
|
||||
# service.
|
||||
async with open_sample_stream(
|
||||
period_s=float(delay_s),
|
||||
loglevel=loglevel,
|
||||
) as istream:
|
||||
) -> tuple[TaskTracker, int]:
|
||||
# TODO: adopt an incremental update engine/approach
|
||||
# where possible here eventually!
|
||||
log.info(f're-syncing fsp {func_name} to source')
|
||||
tracker.cs.cancel()
|
||||
await tracker.complete.wait()
|
||||
tracker, index = await n.start(fsp_target)
|
||||
|
||||
profiler(f'{func_name}: sample stream up')
|
||||
profiler.finish()
|
||||
# always trigger UI refresh after history update,
|
||||
# see ``piker.ui._fsp.FspAdmin.open_chain()`` and
|
||||
# ``piker.ui._display.trigger_update()``.
|
||||
await client_stream.send({
|
||||
'fsp_update': {
|
||||
'key': dst_shm_token,
|
||||
'first': dst._first.value,
|
||||
'last': dst._last.value,
|
||||
}
|
||||
})
|
||||
return tracker, index
|
||||
|
||||
async for i in istream:
|
||||
# print(f'FSP incrementing {i}')
|
||||
def is_synced(
|
||||
src: ShmArray,
|
||||
dst: ShmArray
|
||||
) -> tuple[bool, int, int]:
|
||||
'''
|
||||
Predicate to dertmine if a destination FSP
|
||||
output array is aligned to its source array.
|
||||
|
||||
# respawn the compute task if the source
|
||||
# array has been updated such that we compute
|
||||
# new history from the (prepended) source.
|
||||
synced, step_diff, _ = casc.is_synced()
|
||||
if not synced:
|
||||
step_diff: int = await casc.poll_and_sync_to_step()
|
||||
'''
|
||||
step_diff = src.index - dst.index
|
||||
len_diff = abs(len(src.array) - len(dst.array))
|
||||
return not (
|
||||
# the source is likely backfilling and we must
|
||||
# sync history calculations
|
||||
len_diff > 2
|
||||
|
||||
# skip adding a last bar since we should already
|
||||
# be step alinged
|
||||
if step_diff == 0:
|
||||
continue
|
||||
# we aren't step synced to the source and may be
|
||||
# leading/lagging by a step
|
||||
or step_diff > 1
|
||||
or step_diff < 0
|
||||
), step_diff, len_diff
|
||||
|
||||
# read out last shm row, copy and write new row
|
||||
array = dst_shm.array
|
||||
async def poll_and_sync_to_step(
|
||||
tracker: TaskTracker,
|
||||
src: ShmArray,
|
||||
dst: ShmArray,
|
||||
|
||||
# some metrics like vlm should be reset
|
||||
# to zero every step.
|
||||
if zero_on_step:
|
||||
last = zeroed
|
||||
else:
|
||||
last = array[-1:].copy()
|
||||
) -> tuple[TaskTracker, int]:
|
||||
|
||||
dst.rt_shm.push(last)
|
||||
synced, step_diff, _ = is_synced(src, dst)
|
||||
while not synced:
|
||||
tracker, index = await resync(tracker)
|
||||
synced, step_diff, _ = is_synced(src, dst)
|
||||
|
||||
# sync with source buffer's time step
|
||||
src_l2 = src_shm.array[-2:]
|
||||
src_li, src_lt = src_l2[-1][['index', 'time']]
|
||||
src_2li, src_2lt = src_l2[-2][['index', 'time']]
|
||||
dst_shm._array['time'][src_li] = src_lt
|
||||
dst_shm._array['time'][src_2li] = src_2lt
|
||||
return tracker, step_diff
|
||||
|
||||
# last2 = dst.array[-2:]
|
||||
# if (
|
||||
# last2[-1]['index'] != src_li
|
||||
# or last2[-2]['index'] != src_2li
|
||||
# ):
|
||||
# dstl2 = list(last2)
|
||||
# srcl2 = list(src_l2)
|
||||
# print(
|
||||
# # f'{dst.token}\n'
|
||||
# f'src: {srcl2}\n'
|
||||
# f'dst: {dstl2}\n'
|
||||
# )
|
||||
s, step, ld = is_synced(src, dst)
|
||||
|
||||
# detect sample period step for subscription to increment
|
||||
# signal
|
||||
times = src.array['time']
|
||||
if len(times) > 1:
|
||||
last_ts = times[-1]
|
||||
delay_s = float(last_ts - times[times != last_ts][-1])
|
||||
else:
|
||||
# our default "HFT" sample rate.
|
||||
delay_s = _default_delay_s
|
||||
|
||||
# sub and increment the underlying shared memory buffer
|
||||
# on every step msg received from the global `samplerd`
|
||||
# service.
|
||||
async with open_sample_stream(float(delay_s)) as istream:
|
||||
|
||||
profiler(f'{func_name}: sample stream up')
|
||||
profiler.finish()
|
||||
|
||||
async for i in istream:
|
||||
# print(f'FSP incrementing {i}')
|
||||
|
||||
# respawn the compute task if the source
|
||||
# array has been updated such that we compute
|
||||
# new history from the (prepended) source.
|
||||
synced, step_diff, _ = is_synced(src, dst)
|
||||
if not synced:
|
||||
tracker, step_diff = await poll_and_sync_to_step(
|
||||
tracker,
|
||||
src,
|
||||
dst,
|
||||
)
|
||||
|
||||
# skip adding a last bar since we should already
|
||||
# be step alinged
|
||||
if step_diff == 0:
|
||||
continue
|
||||
|
||||
# read out last shm row, copy and write new row
|
||||
array = dst.array
|
||||
|
||||
# some metrics like vlm should be reset
|
||||
# to zero every step.
|
||||
if zero_on_step:
|
||||
last = zeroed
|
||||
else:
|
||||
last = array[-1:].copy()
|
||||
|
||||
dst.push(last)
|
||||
|
||||
# sync with source buffer's time step
|
||||
src_l2 = src.array[-2:]
|
||||
src_li, src_lt = src_l2[-1][['index', 'time']]
|
||||
src_2li, src_2lt = src_l2[-2][['index', 'time']]
|
||||
dst._array['time'][src_li] = src_lt
|
||||
dst._array['time'][src_2li] = src_2lt
|
||||
|
||||
# last2 = dst.array[-2:]
|
||||
# if (
|
||||
# last2[-1]['index'] != src_li
|
||||
# or last2[-2]['index'] != src_2li
|
||||
# ):
|
||||
# dstl2 = list(last2)
|
||||
# srcl2 = list(src_l2)
|
||||
# print(
|
||||
# # f'{dst.token}\n'
|
||||
# f'src: {srcl2}\n'
|
||||
# f'dst: {dstl2}\n'
|
||||
# )
|
||||
|
|
|
|||
101
piker/log.py
101
piker/log.py
|
|
@ -19,10 +19,6 @@ Log like a forester!
|
|||
"""
|
||||
import logging
|
||||
import json
|
||||
import reprlib
|
||||
from typing import (
|
||||
Callable,
|
||||
)
|
||||
|
||||
import tractor
|
||||
from pygments import (
|
||||
|
|
@ -37,84 +33,35 @@ _proj_name: str = 'piker'
|
|||
|
||||
|
||||
def get_logger(
|
||||
name: str|None = None,
|
||||
**tractor_log_kwargs,
|
||||
name: str = None,
|
||||
|
||||
) -> logging.Logger:
|
||||
'''
|
||||
Return the package log or a sub-logger if a `name=` is provided,
|
||||
which defaults to the calling module's pkg-namespace path.
|
||||
|
||||
See `tractor.log.get_logger()` for details.
|
||||
Return the package log or a sub-log for `name` if provided.
|
||||
|
||||
'''
|
||||
pkg_name: str = _proj_name
|
||||
if (
|
||||
name
|
||||
and
|
||||
pkg_name in name
|
||||
):
|
||||
name: str = name.lstrip(f'{_proj_name}.')
|
||||
|
||||
return tractor.log.get_logger(
|
||||
name=name,
|
||||
pkg_name=pkg_name,
|
||||
**tractor_log_kwargs,
|
||||
_root_name=_proj_name,
|
||||
)
|
||||
|
||||
|
||||
def get_console_log(
|
||||
level: str|None = None,
|
||||
name: str|None = None,
|
||||
pkg_name: str|None = None,
|
||||
with_tractor_log: bool = False,
|
||||
# ?TODO, support a "log-spec" style `str|dict[str, str]` which
|
||||
# dictates both the sublogger-key and a level?
|
||||
# -> see similar idea in `modden`'s usage.
|
||||
**tractor_log_kwargs,
|
||||
level: str | None = None,
|
||||
name: str | None = None,
|
||||
|
||||
) -> logging.Logger:
|
||||
'''
|
||||
Get the package logger and enable a handler which writes to
|
||||
stderr.
|
||||
Get the package logger and enable a handler which writes to stderr.
|
||||
|
||||
Yeah yeah, i know we can use `DictConfig`.
|
||||
You do it.. Bp
|
||||
Yeah yeah, i know we can use ``DictConfig``. You do it...
|
||||
|
||||
'''
|
||||
pkg_name: str = _proj_name
|
||||
if (
|
||||
name
|
||||
and
|
||||
pkg_name in name
|
||||
):
|
||||
name: str = name.lstrip(f'{_proj_name}.')
|
||||
|
||||
tll: str|None = None
|
||||
if (
|
||||
with_tractor_log is not False
|
||||
):
|
||||
tll = level
|
||||
|
||||
elif maybe_actor := tractor.current_actor(
|
||||
err_on_no_runtime=False,
|
||||
):
|
||||
tll = maybe_actor.loglevel
|
||||
|
||||
if tll:
|
||||
t_log = tractor.log.get_console_log(
|
||||
level=tll,
|
||||
name='tractor', # <- XXX, force root tractor log!
|
||||
**tractor_log_kwargs,
|
||||
)
|
||||
# TODO/ allow only enabling certain tractor sub-logs?
|
||||
assert t_log.name == 'tractor'
|
||||
|
||||
return tractor.log.get_console_log(
|
||||
level=level,
|
||||
level,
|
||||
name=name,
|
||||
pkg_name=pkg_name,
|
||||
**tractor_log_kwargs,
|
||||
)
|
||||
_root_name=_proj_name,
|
||||
) # our root logger
|
||||
|
||||
|
||||
def colorize_json(
|
||||
|
|
@ -137,29 +84,3 @@ def colorize_json(
|
|||
# likeable styles: algol_nu, tango, monokai
|
||||
formatters.TerminalTrueColorFormatter(style=style)
|
||||
)
|
||||
|
||||
|
||||
# TODO, eventually defer to the version in `modden` once
|
||||
# it becomes a dep!
|
||||
def mk_repr(
|
||||
**repr_kws,
|
||||
) -> Callable[[str], str]:
|
||||
'''
|
||||
Allocate and deliver a `repr.Repr` instance with provided input
|
||||
settings using the std-lib's `reprlib` mod,
|
||||
* https://docs.python.org/3/library/reprlib.html
|
||||
|
||||
------ Ex. ------
|
||||
An up to 6-layer-nested `dict` as multi-line:
|
||||
- https://stackoverflow.com/a/79102479
|
||||
- https://docs.python.org/3/library/reprlib.html#reprlib.Repr.maxlevel
|
||||
|
||||
'''
|
||||
def_kws: dict[str, int] = dict(
|
||||
indent=2,
|
||||
maxlevel=6, # recursion levels
|
||||
maxstring=66, # match editor line-len limit
|
||||
)
|
||||
def_kws |= repr_kws
|
||||
reprr = reprlib.Repr(**def_kws)
|
||||
return reprr.repr
|
||||
|
|
|
|||
|
|
@ -14,45 +14,49 @@
|
|||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
Actor runtime primtives and (distributed) service APIs for,
|
||||
"""
|
||||
Actor-runtime service orchestration machinery.
|
||||
|
||||
- daemon-service mgmt: `_daemon` (i.e. low-level spawn and supervise machinery
|
||||
for sub-actors like `brokerd`, `emsd`, datad`, etc.)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
- service-actor supervision (via `trio` tasks) API: `._mngr`
|
||||
|
||||
- discovery interface (via light wrapping around `tractor`'s built-in
|
||||
prot): `._registry`
|
||||
|
||||
- `docker` cntr SC supervision for use with `trio`: `_ahab`
|
||||
- wrappers for marketstore and elasticsearch dbs
|
||||
=> TODO: maybe to (re)move elsewhere?
|
||||
|
||||
'''
|
||||
from ._mngr import Services as Services
|
||||
from ._registry import (
|
||||
_tractor_kwargs as _tractor_kwargs,
|
||||
_default_reg_addr as _default_reg_addr,
|
||||
_default_registry_host as _default_registry_host,
|
||||
_default_registry_port as _default_registry_port,
|
||||
|
||||
open_registry as open_registry,
|
||||
find_service as find_service,
|
||||
check_for_service as check_for_service,
|
||||
from ._mngr import Services
|
||||
from ._registry import ( # noqa
|
||||
_tractor_kwargs,
|
||||
_default_reg_addr,
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
open_registry,
|
||||
find_service,
|
||||
check_for_service,
|
||||
)
|
||||
from ._daemon import (
|
||||
maybe_spawn_daemon as maybe_spawn_daemon,
|
||||
spawn_emsd as spawn_emsd,
|
||||
maybe_open_emsd as maybe_open_emsd,
|
||||
from ._daemon import ( # noqa
|
||||
maybe_spawn_daemon,
|
||||
spawn_emsd,
|
||||
maybe_open_emsd,
|
||||
)
|
||||
from ._actor_runtime import (
|
||||
open_piker_runtime as open_piker_runtime,
|
||||
maybe_open_pikerd as maybe_open_pikerd,
|
||||
open_pikerd as open_pikerd,
|
||||
get_runtime_vars as get_runtime_vars,
|
||||
open_piker_runtime,
|
||||
maybe_open_pikerd,
|
||||
open_pikerd,
|
||||
get_tractor_runtime_kwargs,
|
||||
)
|
||||
from ..brokers._daemon import (
|
||||
spawn_brokerd as spawn_brokerd,
|
||||
maybe_spawn_brokerd as maybe_spawn_brokerd,
|
||||
spawn_brokerd,
|
||||
maybe_spawn_brokerd,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
'check_for_service',
|
||||
'Services',
|
||||
'maybe_spawn_daemon',
|
||||
'spawn_brokerd',
|
||||
'maybe_spawn_brokerd',
|
||||
'spawn_emsd',
|
||||
'maybe_open_emsd',
|
||||
'open_piker_runtime',
|
||||
'maybe_open_pikerd',
|
||||
'open_pikerd',
|
||||
'get_tractor_runtime_kwargs',
|
||||
]
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@
|
|||
from __future__ import annotations
|
||||
import os
|
||||
from typing import (
|
||||
Optional,
|
||||
Any,
|
||||
ClassVar,
|
||||
)
|
||||
|
|
@ -31,11 +32,8 @@ from contextlib import (
|
|||
import tractor
|
||||
import trio
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
)
|
||||
from ._util import (
|
||||
subsys,
|
||||
get_console_log,
|
||||
)
|
||||
from ._mngr import (
|
||||
Services,
|
||||
|
|
@ -47,7 +45,7 @@ from ._registry import ( # noqa
|
|||
)
|
||||
|
||||
|
||||
def get_runtime_vars() -> dict[str, Any]:
|
||||
def get_tractor_runtime_kwargs() -> dict[str, Any]:
|
||||
'''
|
||||
Deliver ``tractor`` related runtime variables in a `dict`.
|
||||
|
||||
|
|
@ -58,25 +56,25 @@ def get_runtime_vars() -> dict[str, Any]:
|
|||
@acm
|
||||
async def open_piker_runtime(
|
||||
name: str,
|
||||
registry_addrs: list[tuple[str, int]] = [],
|
||||
|
||||
enable_modules: list[str] = [],
|
||||
loglevel: str|None = None,
|
||||
loglevel: Optional[str] = None,
|
||||
|
||||
# XXX NOTE XXX: you should pretty much never want debug mode
|
||||
# for data daemons when running in production.
|
||||
debug_mode: bool = False,
|
||||
|
||||
registry_addr: None | tuple[str, int] = None,
|
||||
|
||||
# TODO: once we have `rsyscall` support we will read a config
|
||||
# and spawn the service tree distributed per that.
|
||||
start_method: str = 'trio',
|
||||
|
||||
tractor_runtime_overrides: dict|None = None,
|
||||
tractor_runtime_overrides: dict | None = None,
|
||||
**tractor_kwargs,
|
||||
|
||||
) -> tuple[
|
||||
tractor.Actor,
|
||||
list[tuple[str, int]],
|
||||
tuple[str, int],
|
||||
]:
|
||||
'''
|
||||
Start a piker actor who's runtime will automatically sync with
|
||||
|
|
@ -86,72 +84,50 @@ async def open_piker_runtime(
|
|||
a root actor.
|
||||
|
||||
'''
|
||||
# check for existing runtime, boot it
|
||||
# if not already running.
|
||||
try:
|
||||
actor = tractor.current_actor()
|
||||
# check for existing runtime
|
||||
actor = tractor.current_actor().uid
|
||||
|
||||
except tractor._exceptions.NoRuntime:
|
||||
tractor._state._runtime_vars[
|
||||
'piker_vars'
|
||||
] = tractor_runtime_overrides
|
||||
'piker_vars'] = tractor_runtime_overrides
|
||||
|
||||
# NOTE: if no registrar list passed used the default of just
|
||||
# setting it as the root actor on localhost.
|
||||
registry_addrs = (
|
||||
registry_addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
)
|
||||
|
||||
if ems := tractor_kwargs.pop('enable_modules', None):
|
||||
# import pdbp; pdbp.set_trace()
|
||||
enable_modules.extend(ems)
|
||||
registry_addr = registry_addr or _default_reg_addr
|
||||
|
||||
async with (
|
||||
tractor.open_root_actor(
|
||||
|
||||
# passed through to `open_root_actor`
|
||||
registry_addrs=registry_addrs,
|
||||
# passed through to ``open_root_actor``
|
||||
arbiter_addr=registry_addr,
|
||||
name=name,
|
||||
start_method=start_method,
|
||||
loglevel=loglevel,
|
||||
debug_mode=debug_mode,
|
||||
|
||||
# XXX NOTE MEMBER DAT der's a perf hit yo!!
|
||||
# https://greenback.readthedocs.io/en/latest/principle.html#performance
|
||||
maybe_enable_greenback=True,
|
||||
start_method=start_method,
|
||||
|
||||
# TODO: eventually we should be able to avoid
|
||||
# having the root have more then permissions to
|
||||
# spawn other specialized daemons I think?
|
||||
enable_modules=enable_modules,
|
||||
hide_tb=False,
|
||||
|
||||
**tractor_kwargs,
|
||||
) as actor,
|
||||
) as _,
|
||||
|
||||
open_registry(
|
||||
registry_addrs,
|
||||
ensure_exists=False,
|
||||
) as addrs,
|
||||
open_registry(registry_addr, ensure_exists=False) as addr,
|
||||
):
|
||||
assert actor is tractor.current_actor()
|
||||
yield (
|
||||
actor,
|
||||
addrs,
|
||||
tractor.current_actor(),
|
||||
addr,
|
||||
)
|
||||
else:
|
||||
async with open_registry(
|
||||
registry_addrs
|
||||
) as addrs:
|
||||
async with open_registry(registry_addr) as addr:
|
||||
yield (
|
||||
actor,
|
||||
addrs,
|
||||
addr,
|
||||
)
|
||||
|
||||
|
||||
_root_dname: str = 'pikerd'
|
||||
_root_modules: list[str] = [
|
||||
_root_dname = 'pikerd'
|
||||
_root_modules = [
|
||||
__name__,
|
||||
'piker.service._daemon',
|
||||
'piker.brokers._daemon',
|
||||
|
|
@ -165,12 +141,13 @@ _root_modules: list[str] = [
|
|||
|
||||
@acm
|
||||
async def open_pikerd(
|
||||
registry_addrs: list[tuple[str, int]],
|
||||
loglevel: str|None = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
|
||||
# XXX: you should pretty much never want debug mode
|
||||
# for data daemons when running in production.
|
||||
debug_mode: bool = False,
|
||||
registry_addr: None | tuple[str, int] = None,
|
||||
|
||||
**kwargs,
|
||||
|
||||
|
|
@ -182,43 +159,33 @@ async def open_pikerd(
|
|||
alive underling services (see below).
|
||||
|
||||
'''
|
||||
# NOTE: for the root daemon we always enable the root
|
||||
# mod set and we `list.extend()` it into wtv the
|
||||
# caller requested.
|
||||
# TODO: make this mod set more strict?
|
||||
# -[ ] eventually we should be able to avoid
|
||||
# having the root have more then permissions to spawn other
|
||||
# specialized daemons I think?
|
||||
ems: list[str] = kwargs.setdefault('enable_modules', [])
|
||||
ems.extend(_root_modules)
|
||||
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
|
||||
name=_root_dname,
|
||||
# TODO: eventually we should be able to avoid
|
||||
# having the root have more then permissions to
|
||||
# spawn other specialized daemons I think?
|
||||
enable_modules=_root_modules,
|
||||
loglevel=loglevel,
|
||||
debug_mode=debug_mode,
|
||||
registry_addrs=registry_addrs,
|
||||
registry_addr=registry_addr,
|
||||
|
||||
**kwargs,
|
||||
|
||||
) as (
|
||||
root_actor,
|
||||
reg_addrs,
|
||||
),
|
||||
) as (root_actor, reg_addr),
|
||||
tractor.open_nursery() as actor_nursery,
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as service_tn,
|
||||
trio.open_nursery() as service_nursery,
|
||||
):
|
||||
for addr in reg_addrs:
|
||||
if addr not in root_actor.accept_addrs:
|
||||
raise RuntimeError(
|
||||
f'`pikerd` failed to bind on {addr}!\n'
|
||||
'Maybe you have another daemon already running?'
|
||||
)
|
||||
if root_actor.accept_addr != reg_addr:
|
||||
raise RuntimeError(
|
||||
f'`pikerd` failed to bind on {reg_addr}!\n'
|
||||
'Maybe you have another daemon already running?'
|
||||
)
|
||||
|
||||
# assign globally for future daemon/task creation
|
||||
Services.actor_n = actor_nursery
|
||||
Services.service_n = service_tn
|
||||
Services.service_n = service_nursery
|
||||
Services.debug_mode = debug_mode
|
||||
|
||||
try:
|
||||
|
|
@ -228,7 +195,7 @@ async def open_pikerd(
|
|||
# TODO: is this more clever/efficient?
|
||||
# if 'samplerd' in Services.service_tasks:
|
||||
# await Services.cancel_service('samplerd')
|
||||
service_tn.cancel_scope.cancel()
|
||||
service_nursery.cancel_scope.cancel()
|
||||
|
||||
|
||||
# TODO: do we even need this?
|
||||
|
|
@ -258,15 +225,12 @@ async def open_pikerd(
|
|||
|
||||
@acm
|
||||
async def maybe_open_pikerd(
|
||||
registry_addrs: list[tuple[str, int]] | None = None,
|
||||
loglevel: Optional[str] = None,
|
||||
registry_addr: None | tuple = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
**kwargs,
|
||||
|
||||
) -> (
|
||||
tractor._portal.Portal
|
||||
|ClassVar[Services]
|
||||
):
|
||||
) -> tractor._portal.Portal | ClassVar[Services]:
|
||||
'''
|
||||
If no ``pikerd`` daemon-root-actor can be found start it and
|
||||
yield up (we should probably figure out returning a portal to self
|
||||
|
|
@ -274,10 +238,7 @@ async def maybe_open_pikerd(
|
|||
|
||||
'''
|
||||
if loglevel:
|
||||
get_console_log(
|
||||
name=subsys,
|
||||
level=loglevel
|
||||
)
|
||||
get_console_log(loglevel)
|
||||
|
||||
# subtle, we must have the runtime up here or portal lookup will fail
|
||||
query_name = kwargs.pop(
|
||||
|
|
@ -292,52 +253,32 @@ async def maybe_open_pikerd(
|
|||
# async with open_portal(chan) as arb_portal:
|
||||
# yield arb_portal
|
||||
|
||||
registry_addrs: list[tuple[str, int]] = (
|
||||
registry_addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
)
|
||||
|
||||
pikerd_portal: tractor.Portal|None
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
name=query_name,
|
||||
registry_addrs=registry_addrs,
|
||||
registry_addr=registry_addr,
|
||||
loglevel=loglevel,
|
||||
**kwargs,
|
||||
) as (actor, addrs),
|
||||
) as _,
|
||||
|
||||
tractor.find_actor(
|
||||
_root_dname,
|
||||
arbiter_sockaddr=registry_addr,
|
||||
) as portal
|
||||
):
|
||||
if _root_dname in actor.uid:
|
||||
yield None
|
||||
# connect to any existing daemon presuming
|
||||
# its registry socket was selected.
|
||||
if (
|
||||
portal is not None
|
||||
):
|
||||
yield portal
|
||||
return
|
||||
|
||||
# NOTE: IFF running in disti mode, try to attach to any
|
||||
# existing (host-local) `pikerd`.
|
||||
else:
|
||||
async with tractor.find_actor(
|
||||
_root_dname,
|
||||
registry_addrs=registry_addrs,
|
||||
only_first=True,
|
||||
# raise_on_none=True,
|
||||
) as pikerd_portal:
|
||||
|
||||
# connect to any existing remote daemon presuming its
|
||||
# registry socket was selected.
|
||||
if pikerd_portal is not None:
|
||||
|
||||
# sanity check that we are actually connecting to
|
||||
# a remote process and not ourselves.
|
||||
assert actor.uid != pikerd_portal.channel.uid
|
||||
assert registry_addrs
|
||||
|
||||
yield pikerd_portal
|
||||
return
|
||||
|
||||
# presume pikerd role since no daemon could be found at
|
||||
# configured address
|
||||
async with open_pikerd(
|
||||
loglevel=loglevel,
|
||||
registry_addrs=registry_addrs,
|
||||
registry_addr=registry_addr,
|
||||
|
||||
# passthrough to ``tractor`` init
|
||||
**kwargs,
|
||||
|
|
|
|||
|
|
@ -15,8 +15,8 @@
|
|||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
'''
|
||||
Supervisor for ``docker`` with included async and SC wrapping to
|
||||
ensure a cancellable container lifetime system.
|
||||
Supervisor for ``docker`` with included async and SC wrapping
|
||||
to ensure a cancellable container lifetime system.
|
||||
|
||||
'''
|
||||
from __future__ import annotations
|
||||
|
|
@ -49,15 +49,13 @@ from requests.exceptions import (
|
|||
ReadTimeout,
|
||||
)
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
from ._mngr import Services
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
)
|
||||
from .. import config
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
class DockerNotStarted(Exception):
|
||||
'Prolly you dint start da daemon bruh'
|
||||
|
|
@ -338,16 +336,13 @@ class Container:
|
|||
async def open_ahabd(
|
||||
ctx: tractor.Context,
|
||||
endpoint: str, # ns-pointer str-msg-type
|
||||
loglevel: str = 'cancel',
|
||||
loglevel: str | None = None,
|
||||
|
||||
**ep_kwargs,
|
||||
|
||||
) -> None:
|
||||
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
name='piker.service',
|
||||
)
|
||||
log = get_console_log(loglevel or 'cancel')
|
||||
|
||||
async with open_docker() as client:
|
||||
|
||||
|
|
|
|||
|
|
@ -28,11 +28,9 @@ from contextlib import (
|
|||
)
|
||||
|
||||
import tractor
|
||||
from trio.lowlevel import current_task
|
||||
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
)
|
||||
from ._mngr import (
|
||||
Services,
|
||||
|
|
@ -40,17 +38,16 @@ from ._mngr import (
|
|||
from ._actor_runtime import maybe_open_pikerd
|
||||
from ._registry import find_service
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
@acm
|
||||
async def maybe_spawn_daemon(
|
||||
|
||||
service_name: str,
|
||||
service_task_target: Callable,
|
||||
|
||||
spawn_args: dict[str, Any],
|
||||
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
singleton: bool = False,
|
||||
|
||||
**pikerd_kwargs,
|
||||
|
|
@ -68,99 +65,76 @@ async def maybe_spawn_daemon(
|
|||
clients.
|
||||
|
||||
'''
|
||||
log = get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
assert log.name == 'piker.service'
|
||||
|
||||
# serialize access to this section to avoid
|
||||
# 2 or more tasks racing to create a daemon
|
||||
lock = Services.locks[service_name]
|
||||
await lock.acquire()
|
||||
|
||||
try:
|
||||
async with find_service(
|
||||
service_name,
|
||||
registry_addrs=[('127.0.0.1', 6116)],
|
||||
) as portal:
|
||||
if portal is not None:
|
||||
lock.release()
|
||||
yield portal
|
||||
return
|
||||
|
||||
log.warning(
|
||||
f"Couldn't find any existing {service_name}\n"
|
||||
'Attempting to spawn new daemon-service..'
|
||||
)
|
||||
|
||||
# ask root ``pikerd`` daemon to spawn the daemon we need if
|
||||
# pikerd is not live we now become the root of the
|
||||
# process tree
|
||||
async with maybe_open_pikerd(
|
||||
loglevel=loglevel,
|
||||
**pikerd_kwargs,
|
||||
|
||||
) as pikerd_portal:
|
||||
|
||||
# we are the root and thus are `pikerd`
|
||||
# so spawn the target service directly by calling
|
||||
# the provided target routine.
|
||||
# XXX: this assumes that the target is well formed and will
|
||||
# do the right things to setup both a sub-actor **and** call
|
||||
# the ``_Services`` api from above to start the top level
|
||||
# service task for that actor.
|
||||
started: bool
|
||||
if pikerd_portal is None:
|
||||
started = await service_task_target(
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
else:
|
||||
# request a remote `pikerd` (service manager) to start the
|
||||
# target daemon-task, the target can't return
|
||||
# a non-serializable value since it is expected that service
|
||||
# starting is non-blocking and the target task will persist
|
||||
# running "under" or "within" the `pikerd` actor tree after
|
||||
# the questing client disconnects. in other words this
|
||||
# spawns a persistent daemon actor that continues to live
|
||||
# for the lifespan of whatever the service manager inside
|
||||
# `pikerd` says it should.
|
||||
started = await pikerd_portal.run(
|
||||
service_task_target,
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
if started:
|
||||
log.info(f'Service {service_name} started!')
|
||||
|
||||
# block until we can discover (by IPC connection) to the newly
|
||||
# spawned daemon-actor and then deliver the portal to the
|
||||
# caller.
|
||||
async with tractor.wait_for_actor(service_name) as portal:
|
||||
lock.release()
|
||||
yield portal
|
||||
await portal.cancel_actor()
|
||||
|
||||
except BaseException as _err:
|
||||
err = _err
|
||||
if (
|
||||
lock.locked()
|
||||
and
|
||||
lock.statistics().owner is current_task()
|
||||
):
|
||||
log.exception(
|
||||
f'Releasing stale lock after crash..?'
|
||||
f'{err!r}\n'
|
||||
)
|
||||
async with find_service(service_name) as portal:
|
||||
if portal is not None:
|
||||
lock.release()
|
||||
raise err
|
||||
yield portal
|
||||
return
|
||||
|
||||
log.warning(
|
||||
f"Couldn't find any existing {service_name}\n"
|
||||
'Attempting to spawn new daemon-service..'
|
||||
)
|
||||
|
||||
# ask root ``pikerd`` daemon to spawn the daemon we need if
|
||||
# pikerd is not live we now become the root of the
|
||||
# process tree
|
||||
async with maybe_open_pikerd(
|
||||
loglevel=loglevel,
|
||||
**pikerd_kwargs,
|
||||
|
||||
) as pikerd_portal:
|
||||
|
||||
# we are the root and thus are `pikerd`
|
||||
# so spawn the target service directly by calling
|
||||
# the provided target routine.
|
||||
# XXX: this assumes that the target is well formed and will
|
||||
# do the right things to setup both a sub-actor **and** call
|
||||
# the ``_Services`` api from above to start the top level
|
||||
# service task for that actor.
|
||||
started: bool
|
||||
if pikerd_portal is None:
|
||||
started = await service_task_target(
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
else:
|
||||
# request a remote `pikerd` (service manager) to start the
|
||||
# target daemon-task, the target can't return
|
||||
# a non-serializable value since it is expected that service
|
||||
# starting is non-blocking and the target task will persist
|
||||
# running "under" or "within" the `pikerd` actor tree after
|
||||
# the questing client disconnects. in other words this
|
||||
# spawns a persistent daemon actor that continues to live
|
||||
# for the lifespan of whatever the service manager inside
|
||||
# `pikerd` says it should.
|
||||
started = await pikerd_portal.run(
|
||||
service_task_target,
|
||||
loglevel=loglevel,
|
||||
**spawn_args,
|
||||
)
|
||||
|
||||
if started:
|
||||
log.info(f'Service {service_name} started!')
|
||||
|
||||
# block until we can discover (by IPC connection) to the newly
|
||||
# spawned daemon-actor and then deliver the portal to the
|
||||
# caller.
|
||||
async with tractor.wait_for_actor(service_name) as portal:
|
||||
lock.release()
|
||||
yield portal
|
||||
await portal.cancel_actor()
|
||||
|
||||
|
||||
async def spawn_emsd(
|
||||
loglevel: str|None = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
**extra_tractor_kwargs
|
||||
|
||||
) -> bool:
|
||||
|
|
@ -197,8 +171,9 @@ async def spawn_emsd(
|
|||
|
||||
@acm
|
||||
async def maybe_open_emsd(
|
||||
|
||||
brokername: str,
|
||||
loglevel: str|None = None,
|
||||
loglevel: str | None = None,
|
||||
|
||||
**pikerd_kwargs,
|
||||
|
||||
|
|
|
|||
|
|
@ -27,25 +27,17 @@ from typing import (
|
|||
import trio
|
||||
from trio_typing import TaskStatus
|
||||
import tractor
|
||||
from tractor import (
|
||||
current_actor,
|
||||
ContextCancelled,
|
||||
Context,
|
||||
Portal,
|
||||
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
)
|
||||
|
||||
from piker.log import get_logger
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
# TODO: we need remote wrapping and a general soln:
|
||||
# - factor this into a ``tractor.highlevel`` extension # pack for the
|
||||
# library.
|
||||
# - wrap a "remote api" wherein you can get a method proxy
|
||||
# to the pikerd actor for starting services remotely!
|
||||
# - prolly rename this to ActorServicesNursery since it spawns
|
||||
# new actors and supervises them to completion?
|
||||
class Services:
|
||||
|
||||
actor_n: tractor._supervise.ActorNursery
|
||||
|
|
@ -55,7 +47,7 @@ class Services:
|
|||
str,
|
||||
tuple[
|
||||
trio.CancelScope,
|
||||
Portal,
|
||||
tractor.Portal,
|
||||
trio.Event,
|
||||
]
|
||||
] = {}
|
||||
|
|
@ -65,12 +57,12 @@ class Services:
|
|||
async def start_service_task(
|
||||
self,
|
||||
name: str,
|
||||
portal: Portal,
|
||||
portal: tractor.Portal,
|
||||
target: Callable,
|
||||
allow_overruns: bool = False,
|
||||
**ctx_kwargs,
|
||||
|
||||
) -> (trio.CancelScope, Context):
|
||||
) -> (trio.CancelScope, tractor.Context):
|
||||
'''
|
||||
Open a context in a service sub-actor, add to a stack
|
||||
that gets unwound at ``pikerd`` teardown.
|
||||
|
|
@ -109,30 +101,13 @@ class Services:
|
|||
# wait on any context's return value
|
||||
# and any final portal result from the
|
||||
# sub-actor.
|
||||
ctx_res: Any = await ctx.wait_for_result()
|
||||
ctx_res = await ctx.result()
|
||||
|
||||
# NOTE: blocks indefinitely until cancelled
|
||||
# either by error from the target context
|
||||
# function or by being cancelled here by the
|
||||
# surrounding cancel scope.
|
||||
return (await portal.result(), ctx_res)
|
||||
except ContextCancelled as ctxe:
|
||||
canceller: tuple[str, str] = ctxe.canceller
|
||||
our_uid: tuple[str, str] = current_actor().uid
|
||||
if (
|
||||
canceller != portal.channel.uid
|
||||
and
|
||||
canceller != our_uid
|
||||
):
|
||||
log.cancel(
|
||||
f'Actor-service {name} was remotely cancelled?\n'
|
||||
f'remote canceller: {canceller}\n'
|
||||
f'Keeping {our_uid} alive, ignoring sub-actor cancel..\n'
|
||||
)
|
||||
else:
|
||||
raise
|
||||
|
||||
|
||||
|
||||
finally:
|
||||
await portal.cancel_actor()
|
||||
|
|
|
|||
|
|
@ -27,29 +27,14 @@ from typing import (
|
|||
)
|
||||
|
||||
import tractor
|
||||
from tractor import (
|
||||
msg,
|
||||
Actor,
|
||||
Portal,
|
||||
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
)
|
||||
|
||||
from piker.log import get_logger
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
# TODO? default path-space for UDS registry?
|
||||
# [ ] needs to be Xplatform tho!
|
||||
# _default_registry_path: Path = (
|
||||
# Path(os.environ['XDG_RUNTIME_DIR'])
|
||||
# /'piker'
|
||||
# )
|
||||
|
||||
_default_registry_host: str = '127.0.0.1'
|
||||
_default_registry_port: int = 6116
|
||||
_default_reg_addr: tuple[
|
||||
str,
|
||||
int, # |str TODO, once we support UDS, see above.
|
||||
] = (
|
||||
_default_reg_addr: tuple[str, int] = (
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
)
|
||||
|
|
@ -61,9 +46,7 @@ _registry: Registry | None = None
|
|||
|
||||
|
||||
class Registry:
|
||||
# TODO: should this be a set or should we complain
|
||||
# on duplicates?
|
||||
addrs: list[tuple[str, int]] = []
|
||||
addr: None | tuple[str, int] = None
|
||||
|
||||
# TODO: table of uids to sockaddrs
|
||||
peers: dict[
|
||||
|
|
@ -77,158 +60,82 @@ _tractor_kwargs: dict[str, Any] = {}
|
|||
|
||||
@acm
|
||||
async def open_registry(
|
||||
addrs: list[tuple[str, int]],
|
||||
addr: None | tuple[str, int] = None,
|
||||
ensure_exists: bool = True,
|
||||
|
||||
) -> list[tuple[str, int]]:
|
||||
'''
|
||||
Open the service-actor-discovery registry by returning a set of
|
||||
tranport socket-addrs to registrar actors which may be
|
||||
contacted and queried for similar addresses for other
|
||||
non-registrar actors.
|
||||
) -> tuple[str, int]:
|
||||
|
||||
'''
|
||||
global _tractor_kwargs
|
||||
actor: Actor = tractor.current_actor()
|
||||
aid: msg.Aid = actor.aid
|
||||
uid: tuple[str, str] = aid.uid
|
||||
preset_reg_addrs: list[
|
||||
tuple[str, int]
|
||||
] = Registry.addrs
|
||||
actor = tractor.current_actor()
|
||||
uid = actor.uid
|
||||
if (
|
||||
preset_reg_addrs
|
||||
and
|
||||
addrs
|
||||
Registry.addr is not None
|
||||
and addr
|
||||
):
|
||||
if preset_reg_addrs != addrs:
|
||||
# if any(addr in preset_reg_addrs for addr in addrs):
|
||||
diff: set[
|
||||
tuple[str, int]
|
||||
] = set(preset_reg_addrs) - set(addrs)
|
||||
if diff:
|
||||
log.warning(
|
||||
f'`{uid}` requested only subset of registrars: {addrs}\n'
|
||||
f'However there are more @{diff}'
|
||||
)
|
||||
else:
|
||||
raise RuntimeError(
|
||||
f'`{uid}` has non-matching registrar addresses?\n'
|
||||
f'request: {addrs}\n'
|
||||
f'already set: {preset_reg_addrs}'
|
||||
)
|
||||
raise RuntimeError(
|
||||
f'`{uid}` registry addr already bound @ {_registry.sockaddr}'
|
||||
)
|
||||
|
||||
was_set: bool = False
|
||||
|
||||
if (
|
||||
not tractor.is_root_process()
|
||||
and
|
||||
not Registry.addrs
|
||||
and Registry.addr is None
|
||||
):
|
||||
Registry.addrs.extend(actor.reg_addrs)
|
||||
Registry.addr = actor._arb_addr
|
||||
|
||||
if (
|
||||
ensure_exists
|
||||
and
|
||||
not Registry.addrs
|
||||
and Registry.addr is None
|
||||
):
|
||||
raise RuntimeError(
|
||||
f"`{uid}` registry should already exist but doesn't?"
|
||||
f"`{uid}` registry should already exist bug doesn't?"
|
||||
)
|
||||
|
||||
if not Registry.addrs:
|
||||
if (
|
||||
Registry.addr is None
|
||||
):
|
||||
was_set = True
|
||||
Registry.addrs = (
|
||||
addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
)
|
||||
Registry.addr = addr or _default_reg_addr
|
||||
|
||||
# NOTE: only spot this seems currently used is inside
|
||||
# `.ui._exec` which is the (eventual qtloops) bootstrapping
|
||||
# with guest mode.
|
||||
reg_addrs: list[tuple[str, str|int]] = Registry.addrs
|
||||
# !TODO, a struct-API to stringently allow this only in special
|
||||
# cases?
|
||||
# -> better would be to have some way to (atomically) rewrite
|
||||
# and entire `RuntimeVars`?? ideas welcome obvi..
|
||||
_tractor_kwargs['registry_addrs'] = reg_addrs
|
||||
_tractor_kwargs['arbiter_addr'] = Registry.addr
|
||||
|
||||
try:
|
||||
yield Registry.addrs
|
||||
yield Registry.addr
|
||||
finally:
|
||||
# XXX: always clear the global addr if we set it so that the
|
||||
# next (set of) calls will apply whatever new one is passed
|
||||
# in.
|
||||
if was_set:
|
||||
Registry.addrs = None
|
||||
Registry.addr = None
|
||||
|
||||
|
||||
@acm
|
||||
async def find_service(
|
||||
service_name: str,
|
||||
registry_addrs: list[tuple[str, int]] | None = None,
|
||||
|
||||
first_only: bool = True,
|
||||
|
||||
) -> (
|
||||
Portal
|
||||
| list[Portal]
|
||||
| None
|
||||
):
|
||||
# try:
|
||||
reg_addrs: list[tuple[str, int|str]]
|
||||
async with open_registry(
|
||||
addrs=(
|
||||
registry_addrs
|
||||
# NOTE: if no addr set is passed assume the registry has
|
||||
# already been opened and use the previously applied
|
||||
# startup set.
|
||||
or Registry.addrs
|
||||
),
|
||||
) as reg_addrs:
|
||||
|
||||
log.info(
|
||||
f'Scanning for service {service_name!r}'
|
||||
)
|
||||
) -> tractor.Portal | None:
|
||||
|
||||
async with open_registry() as reg_addr:
|
||||
log.info(f'Scanning for service `{service_name}`')
|
||||
# attach to existing daemon by name if possible
|
||||
maybe_portals: list[Portal]|Portal|None
|
||||
async with tractor.find_actor(
|
||||
service_name,
|
||||
registry_addrs=reg_addrs,
|
||||
only_first=first_only, # if set only returns single ref
|
||||
) as maybe_portals:
|
||||
if not maybe_portals:
|
||||
log.info(
|
||||
f'Could NOT find service {service_name!r} -> {maybe_portals!r}'
|
||||
)
|
||||
yield None
|
||||
return
|
||||
|
||||
log.info(
|
||||
f'Found service {service_name!r} -> {maybe_portals}'
|
||||
)
|
||||
yield maybe_portals
|
||||
|
||||
# except BaseException as _berr:
|
||||
# berr = _berr
|
||||
# log.exception(
|
||||
# 'tractor.find_actor() failed with,\n'
|
||||
# )
|
||||
# raise berr
|
||||
arbiter_sockaddr=reg_addr,
|
||||
) as maybe_portal:
|
||||
yield maybe_portal
|
||||
|
||||
|
||||
async def check_for_service(
|
||||
service_name: str,
|
||||
) -> None|tuple[str, int]:
|
||||
|
||||
) -> None | tuple[str, int]:
|
||||
'''
|
||||
Service daemon "liveness" predicate.
|
||||
|
||||
'''
|
||||
async with (
|
||||
open_registry(ensure_exists=False) as reg_addr,
|
||||
tractor.query_actor(
|
||||
async with open_registry(ensure_exists=False) as reg_addr:
|
||||
async with tractor.query_actor(
|
||||
service_name,
|
||||
arbiter_sockaddr=reg_addr,
|
||||
) as sockaddr,
|
||||
):
|
||||
return sockaddr
|
||||
) as sockaddr:
|
||||
return sockaddr
|
||||
|
|
|
|||
|
|
@ -14,12 +14,20 @@
|
|||
# You should have received a copy of the GNU Affero General Public License
|
||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
"""
|
||||
Sub-sys module commons (if any ?? Bp).
|
||||
Sub-sys module commons.
|
||||
|
||||
"""
|
||||
from functools import partial
|
||||
|
||||
from ..log import (
|
||||
get_logger,
|
||||
get_console_log,
|
||||
)
|
||||
subsys: str = 'piker.service'
|
||||
|
||||
# ?TODO, if we were going to keep a `get_console_log()` in here to be
|
||||
# invoked at `import`-time, how do we dynamically hand in the
|
||||
# `level=` value? seems too early in the runtime to be injected
|
||||
# right?
|
||||
log = get_logger(subsys)
|
||||
|
||||
get_console_log = partial(
|
||||
get_console_log,
|
||||
name=subsys,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -16,7 +16,6 @@
|
|||
|
||||
from __future__ import annotations
|
||||
from contextlib import asynccontextmanager as acm
|
||||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
TYPE_CHECKING,
|
||||
|
|
@ -27,17 +26,12 @@ import asks
|
|||
if TYPE_CHECKING:
|
||||
import docker
|
||||
from ._ahab import DockerContainer
|
||||
from . import (
|
||||
Services,
|
||||
)
|
||||
|
||||
from piker.log import (
|
||||
from ._util import log # sub-sys logger
|
||||
from ._util import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
# container level config
|
||||
_config = {
|
||||
|
|
@ -73,10 +67,7 @@ def start_elasticsearch(
|
|||
elastic
|
||||
|
||||
'''
|
||||
get_console_log(
|
||||
level='info',
|
||||
name=__name__,
|
||||
)
|
||||
get_console_log('info', name=__name__)
|
||||
|
||||
dcntr: DockerContainer = client.containers.run(
|
||||
'piker:elastic',
|
||||
|
|
|
|||
|
|
@ -52,18 +52,17 @@ import pendulum
|
|||
# TODO: import this for specific error set expected by mkts client
|
||||
# import purerpc
|
||||
|
||||
from piker.data.feed import maybe_open_feed
|
||||
from ..data.feed import maybe_open_feed
|
||||
from . import Services
|
||||
from piker.log import (
|
||||
from ._util import (
|
||||
log, # sub-sys logger
|
||||
get_console_log,
|
||||
get_logger,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import docker
|
||||
from ._ahab import DockerContainer
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
# ahabd-supervisor and container level config
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@ from typing import (
|
|||
|
||||
import numpy as np
|
||||
|
||||
|
||||
from .. import config
|
||||
from ..service import (
|
||||
check_for_service,
|
||||
|
|
@ -138,23 +139,13 @@ class StorageClient(
|
|||
...
|
||||
|
||||
|
||||
class TimeseriesNotFound(Exception):
|
||||
'''
|
||||
No timeseries entry can be found for this backend.
|
||||
|
||||
'''
|
||||
|
||||
|
||||
class StorageConnectionError(ConnectionError):
|
||||
'''
|
||||
Can't connect to the desired tsdb subsys/service.
|
||||
|
||||
'''
|
||||
|
||||
def get_storagemod(
|
||||
name: str,
|
||||
|
||||
) -> ModuleType:
|
||||
def get_storagemod(name: str) -> ModuleType:
|
||||
mod: ModuleType = import_module(
|
||||
'.' + name,
|
||||
'piker.storage',
|
||||
|
|
@ -167,12 +158,9 @@ def get_storagemod(
|
|||
|
||||
@acm
|
||||
async def open_storage_client(
|
||||
backend: str|None = None,
|
||||
backend: str | None = None,
|
||||
|
||||
) -> tuple[
|
||||
ModuleType,
|
||||
StorageClient,
|
||||
]:
|
||||
) -> tuple[ModuleType, StorageClient]:
|
||||
'''
|
||||
Load the ``StorageClient`` for named backend.
|
||||
|
||||
|
|
@ -181,13 +169,10 @@ async def open_storage_client(
|
|||
tsdb_host: str = 'localhost'
|
||||
|
||||
# load root config and any tsdb user defined settings
|
||||
conf, path = config.load(
|
||||
conf_name='conf',
|
||||
touch_if_dne=True,
|
||||
)
|
||||
conf, path = config.load('conf', touch_if_dne=True)
|
||||
|
||||
# TODO: maybe not under a "network" section.. since
|
||||
# no more chitty `marketstore`..
|
||||
# no more chitty mkts..
|
||||
tsdbconf: dict = {}
|
||||
service_section = conf.get('service')
|
||||
if (
|
||||
|
|
@ -198,11 +183,8 @@ async def open_storage_client(
|
|||
|
||||
# lookup backend tsdb module by name and load any user service
|
||||
# settings for connecting to the tsdb service.
|
||||
backend: str = tsdbconf.pop(
|
||||
'name',
|
||||
def_backend,
|
||||
)
|
||||
tsdb_host: str = tsdbconf.get('maddrs', [])
|
||||
backend: str = tsdbconf.pop('backend')
|
||||
tsdb_host: str = tsdbconf['host']
|
||||
|
||||
if backend is None:
|
||||
backend: str = def_backend
|
||||
|
|
@ -272,10 +254,7 @@ async def open_tsdb_client(
|
|||
from ..data.feed import maybe_open_feed
|
||||
|
||||
async with (
|
||||
open_storage_client() as (
|
||||
_,
|
||||
storage,
|
||||
),
|
||||
open_storage_client() as (_, storage),
|
||||
|
||||
maybe_open_feed(
|
||||
[fqme],
|
||||
|
|
@ -283,7 +262,7 @@ async def open_tsdb_client(
|
|||
|
||||
) as feed,
|
||||
):
|
||||
profiler(f'opened feed for {fqme!r}')
|
||||
profiler(f'opened feed for {fqme}')
|
||||
|
||||
# to_append = feed.hist_shm.array
|
||||
# to_prepend = None
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
# piker: trading gear for hackers
|
||||
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of piker0)
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Affero General Public License as published by
|
||||
|
|
@ -21,10 +21,8 @@ Storage middle-ware CLIs.
|
|||
from __future__ import annotations
|
||||
from pathlib import Path
|
||||
import time
|
||||
from types import ModuleType
|
||||
from typing import (
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
from typing import Generator
|
||||
# from typing import TYPE_CHECKING
|
||||
|
||||
import polars as pl
|
||||
import numpy as np
|
||||
|
|
@ -37,20 +35,24 @@ import typer
|
|||
|
||||
from piker.service import open_piker_runtime
|
||||
from piker.cli import cli
|
||||
from piker.config import get_conf_dir
|
||||
from piker.data import (
|
||||
maybe_open_shm_array,
|
||||
def_iohlcv_fields,
|
||||
ShmArray,
|
||||
)
|
||||
from piker import tsp
|
||||
from . import log
|
||||
from piker.data.history import (
|
||||
_default_hist_size,
|
||||
_default_rt_size,
|
||||
)
|
||||
from . import (
|
||||
log,
|
||||
)
|
||||
from . import (
|
||||
__tsdbs__,
|
||||
open_storage_client,
|
||||
StorageClient,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from piker.ui._remote_ctl import AnnotCtl
|
||||
|
||||
|
||||
store = typer.Typer()
|
||||
|
||||
|
|
@ -75,6 +77,7 @@ def ls(
|
|||
async with (
|
||||
open_piker_runtime(
|
||||
'tsdb_storage',
|
||||
enable_modules=['piker.service._ahab'],
|
||||
),
|
||||
):
|
||||
for i, backend in enumerate(backends):
|
||||
|
|
@ -96,18 +99,6 @@ def ls(
|
|||
trio.run(query_all)
|
||||
|
||||
|
||||
# TODO: like ls but takes in a pattern and matches
|
||||
# @store.command()
|
||||
# def search(
|
||||
# patt: str,
|
||||
# backends: list[str] = typer.Argument(
|
||||
# default=None,
|
||||
# help='Storage backends to query, default is all.'
|
||||
# ),
|
||||
# ):
|
||||
# ...
|
||||
|
||||
|
||||
@store.command()
|
||||
def delete(
|
||||
symbols: list[str],
|
||||
|
|
@ -130,6 +121,7 @@ def delete(
|
|||
async with (
|
||||
open_piker_runtime(
|
||||
'tsdb_storage',
|
||||
enable_modules=['piker.service._ahab']
|
||||
),
|
||||
open_storage_client(backend) as (_, client),
|
||||
trio.open_nursery() as n,
|
||||
|
|
@ -150,33 +142,21 @@ def delete(
|
|||
def anal(
|
||||
fqme: str,
|
||||
period: int = 60,
|
||||
pdb: bool = False,
|
||||
|
||||
) -> np.ndarray:
|
||||
'''
|
||||
Anal-ysis is when you take the data do stuff to it.
|
||||
|
||||
NOTE: This ONLY loads the offline timeseries data (by default
|
||||
from a parquet file) NOT the in-shm version you might be seeing
|
||||
in a chart.
|
||||
|
||||
'''
|
||||
async def main():
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
# are you a bear or boi?
|
||||
'tsdb_polars_anal',
|
||||
debug_mode=pdb,
|
||||
),
|
||||
open_storage_client() as (
|
||||
mod,
|
||||
client,
|
||||
# enable_modules=['piker.service._ahab']
|
||||
debug_mode=True,
|
||||
),
|
||||
open_storage_client() as (mod, client),
|
||||
):
|
||||
syms: list[str] = await client.list_keys()
|
||||
log.info(f'{len(syms)} FOUND for {mod.name}')
|
||||
print(f'{len(syms)} FOUND for {mod.name}')
|
||||
|
||||
history: ShmArray # np buffer format
|
||||
(
|
||||
history,
|
||||
first_dt,
|
||||
|
|
@ -187,292 +167,179 @@ def anal(
|
|||
)
|
||||
assert first_dt < last_dt
|
||||
|
||||
null_segs: tuple = tsp.get_null_segs(
|
||||
frame=history,
|
||||
period=period,
|
||||
)
|
||||
# TODO: do tsp queries to backcend to fill i missing
|
||||
# history and then prolly write it to tsdb!
|
||||
src_df = await client.as_df(fqme, period)
|
||||
from piker.data import _timeseries as tsmod
|
||||
df: pl.DataFrame = tsmod.with_dts(src_df)
|
||||
gaps: pl.DataFrame = tsmod.detect_time_gaps(df)
|
||||
|
||||
shm_df: pl.DataFrame = await client.as_df(
|
||||
fqme,
|
||||
period,
|
||||
)
|
||||
if not gaps.is_empty():
|
||||
print(f'Gaps found:\n{gaps}')
|
||||
|
||||
df: pl.DataFrame # with dts
|
||||
deduped: pl.DataFrame # deduplicated dts
|
||||
(
|
||||
df,
|
||||
deduped,
|
||||
diff,
|
||||
) = tsp.dedupe(
|
||||
shm_df,
|
||||
period=period,
|
||||
)
|
||||
|
||||
write_edits: bool = True
|
||||
if (
|
||||
write_edits
|
||||
and (
|
||||
diff
|
||||
or null_segs
|
||||
)
|
||||
):
|
||||
await tractor.pause()
|
||||
await client.write_ohlcv(
|
||||
fqme,
|
||||
ohlcv=deduped,
|
||||
timeframe=period,
|
||||
)
|
||||
|
||||
else:
|
||||
# TODO: something better with tab completion..
|
||||
# is there something more minimal but nearly as
|
||||
# functional as ipython?
|
||||
await tractor.pause()
|
||||
assert not null_segs
|
||||
# TODO: something better with tab completion..
|
||||
# is there something more minimal but nearly as
|
||||
# functional as ipython?
|
||||
await tractor.pause()
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
def iter_dfs_from_shms(fqme: str) -> Generator[
|
||||
tuple[Path, ShmArray, pl.DataFrame],
|
||||
None,
|
||||
None,
|
||||
]:
|
||||
# shm buffer size table based on known sample rates
|
||||
sizes: dict[str, int] = {
|
||||
'hist': _default_hist_size,
|
||||
'rt': _default_rt_size,
|
||||
}
|
||||
|
||||
# load all detected shm buffer files which have the
|
||||
# passed FQME pattern in the file name.
|
||||
shmfiles: list[Path] = []
|
||||
shmdir = Path('/dev/shm/')
|
||||
|
||||
for shmfile in shmdir.glob(f'*{fqme}*'):
|
||||
filename: str = shmfile.name
|
||||
|
||||
# skip index files
|
||||
if (
|
||||
'_first' in filename
|
||||
or '_last' in filename
|
||||
):
|
||||
continue
|
||||
|
||||
assert shmfile.is_file()
|
||||
log.debug(f'Found matching shm buffer file: {filename}')
|
||||
shmfiles.append(shmfile)
|
||||
|
||||
for shmfile in shmfiles:
|
||||
|
||||
# lookup array buffer size based on file suffix
|
||||
# being either .rt or .hist
|
||||
key: str = shmfile.name.rsplit('.')[-1]
|
||||
|
||||
# skip FSP buffers for now..
|
||||
if key not in sizes:
|
||||
continue
|
||||
|
||||
size: int = sizes[key]
|
||||
|
||||
# attach to any shm buffer, load array into polars df,
|
||||
# write to local parquet file.
|
||||
shm, opened = maybe_open_shm_array(
|
||||
key=shmfile.name,
|
||||
size=size,
|
||||
dtype=def_iohlcv_fields,
|
||||
readonly=True,
|
||||
)
|
||||
assert not opened
|
||||
ohlcv = shm.array
|
||||
|
||||
start = time.time()
|
||||
|
||||
# XXX: thanks to this SO answer for this conversion tip:
|
||||
# https://stackoverflow.com/a/72054819
|
||||
df = pl.DataFrame({
|
||||
field_name: ohlcv[field_name]
|
||||
for field_name in ohlcv.dtype.fields
|
||||
})
|
||||
delay: float = round(
|
||||
time.time() - start,
|
||||
ndigits=6,
|
||||
)
|
||||
log.info(
|
||||
f'numpy -> polars conversion took {delay} secs\n'
|
||||
f'polars df: {df}'
|
||||
)
|
||||
|
||||
yield (
|
||||
shmfile,
|
||||
shm,
|
||||
df,
|
||||
)
|
||||
|
||||
|
||||
@store.command()
|
||||
def ldshm(
|
||||
fqme: str,
|
||||
write_parquet: bool = True,
|
||||
reload_parquet_to_shm: bool = True,
|
||||
pdb: bool = False, # --pdb passed?
|
||||
|
||||
write_parquet: bool = False,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Linux ONLY: load any fqme file name matching shm buffer from
|
||||
/dev/shm/ into an OHLCV numpy array and polars DataFrame,
|
||||
optionally write to offline storage via `.parquet` file.
|
||||
optionally write to .parquet file.
|
||||
|
||||
'''
|
||||
async def main():
|
||||
from piker.ui._remote_ctl import (
|
||||
open_annot_ctl,
|
||||
)
|
||||
actl: AnnotCtl
|
||||
mod: ModuleType
|
||||
client: StorageClient
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
'polars_boi',
|
||||
enable_modules=['piker.data._sharedmem'],
|
||||
debug_mode=pdb,
|
||||
debug_mode=True,
|
||||
),
|
||||
open_storage_client() as (
|
||||
mod,
|
||||
client,
|
||||
),
|
||||
open_annot_ctl() as actl,
|
||||
):
|
||||
shm_df: pl.DataFrame | None = None
|
||||
tf2aids: dict[float, dict] = {}
|
||||
|
||||
for (
|
||||
shmfile,
|
||||
shm,
|
||||
# parquet_path,
|
||||
shm_df,
|
||||
) in tsp.iter_dfs_from_shms(fqme):
|
||||
df: pl.DataFrame | None = None
|
||||
for shmfile, shm, src_df in iter_dfs_from_shms(fqme):
|
||||
|
||||
# compute ohlc properties for naming
|
||||
times: np.ndarray = shm.array['time']
|
||||
d1: float = float(times[-1] - times[-2])
|
||||
d2: float = 0
|
||||
# XXX, take a median sample rate if sufficient data
|
||||
if times.size > 2:
|
||||
d2: float = float(times[-2] - times[-3])
|
||||
med: float = np.median(np.diff(times))
|
||||
if (
|
||||
d1 < 1.
|
||||
and d2 < 1.
|
||||
and med < 1.
|
||||
):
|
||||
raise ValueError(
|
||||
f'Something is wrong with time period for {shm}:\n{times}'
|
||||
)
|
||||
period_s: float = float(max(d1, d2, med))
|
||||
|
||||
null_segs: tuple = tsp.get_null_segs(
|
||||
frame=shm.array,
|
||||
period=period_s,
|
||||
)
|
||||
|
||||
# TODO: call null-seg fixer somehow?
|
||||
if null_segs:
|
||||
|
||||
if tractor._state.is_debug_mode():
|
||||
await tractor.pause()
|
||||
# async with (
|
||||
# trio.open_nursery() as tn,
|
||||
# mod.open_history_client(
|
||||
# mkt,
|
||||
# ) as (get_hist, config),
|
||||
# ):
|
||||
# nulls_detected: trio.Event = await tn.start(partial(
|
||||
# tsp.maybe_fill_null_segments,
|
||||
|
||||
# shm=shm,
|
||||
# timeframe=timeframe,
|
||||
# get_hist=get_hist,
|
||||
# sampler_stream=sampler_stream,
|
||||
# mkt=mkt,
|
||||
# ))
|
||||
|
||||
# over-write back to shm?
|
||||
wdts: pl.DataFrame # with dts
|
||||
deduped: pl.DataFrame # deduplicated dts
|
||||
(
|
||||
wdts,
|
||||
deduped,
|
||||
diff,
|
||||
valid_races,
|
||||
dq_issues,
|
||||
) = tsp.dedupe_ohlcv_smart(
|
||||
shm_df,
|
||||
)
|
||||
|
||||
# Report duplicate analysis
|
||||
if diff > 0:
|
||||
log.info(
|
||||
f'Removed {diff} duplicate timestamp(s)\n'
|
||||
secs: float = times[-1] - times[-2]
|
||||
if secs < 1.:
|
||||
raise ValueError(
|
||||
f'Something is wrong with time period for {shm}:\n{times}'
|
||||
)
|
||||
if valid_races is not None:
|
||||
identical: int = (
|
||||
valid_races
|
||||
.filter(pl.col('identical_bars'))
|
||||
.height
|
||||
)
|
||||
monotonic: int = valid_races.height - identical
|
||||
log.info(
|
||||
f'Valid race conditions: {valid_races.height}\n'
|
||||
f' - Identical bars: {identical}\n'
|
||||
f' - Volume monotonic: {monotonic}\n'
|
||||
)
|
||||
|
||||
if dq_issues is not None:
|
||||
log.warning(
|
||||
f'DATA QUALITY ISSUES from provider: '
|
||||
f'{dq_issues.height} timestamp(s)\n'
|
||||
f'{dq_issues}\n'
|
||||
)
|
||||
from piker.data import _timeseries as tsmod
|
||||
df: pl.DataFrame = tsmod.with_dts(src_df)
|
||||
gaps: pl.DataFrame = tsmod.detect_time_gaps(df)
|
||||
|
||||
# detect gaps from in expected (uniform OHLC) sample period
|
||||
step_gaps: pl.DataFrame = tsp.detect_time_gaps(
|
||||
deduped,
|
||||
expect_period=period_s,
|
||||
)
|
||||
|
||||
# TODO: by default we always want to mark these up
|
||||
# with rects showing up/down gaps Bo
|
||||
venue_gaps: pl.DataFrame = tsp.detect_time_gaps(
|
||||
deduped,
|
||||
expect_period=period_s,
|
||||
|
||||
# TODO: actually pull the exact duration
|
||||
# expected for each venue operational period?
|
||||
# gap_dt_unit='day',
|
||||
gap_dt_unit='day',
|
||||
gap_thresh=1,
|
||||
)
|
||||
|
||||
# TODO: find the disjoint set of step gaps from
|
||||
# venue (closure) set!
|
||||
# -[ ] do a set diff by checking for the unique
|
||||
# gap set only in the step_gaps?
|
||||
# TODO: maybe only optionally enter this depending
|
||||
# on some CLI flags and/or gap detection?
|
||||
if (
|
||||
not venue_gaps.is_empty()
|
||||
or (
|
||||
not step_gaps.is_empty()
|
||||
# XXX, i presume i put this bc i was guarding
|
||||
# for ib venue gaps?
|
||||
# and
|
||||
# period_s < 60
|
||||
)
|
||||
not gaps.is_empty()
|
||||
or secs > 2
|
||||
):
|
||||
# write repaired ts to parquet-file?
|
||||
if write_parquet:
|
||||
start: float = time.time()
|
||||
path: Path = await client.write_ohlcv(
|
||||
fqme,
|
||||
ohlcv=deduped,
|
||||
timeframe=period_s,
|
||||
)
|
||||
write_delay: float = round(
|
||||
time.time() - start,
|
||||
ndigits=6,
|
||||
)
|
||||
await tractor.pause()
|
||||
|
||||
# read back from fs
|
||||
start: float = time.time()
|
||||
read_df: pl.DataFrame = pl.read_parquet(path)
|
||||
read_delay: float = round(
|
||||
time.time() - start,
|
||||
ndigits=6,
|
||||
)
|
||||
log.info(
|
||||
f'parquet write took {write_delay} secs\n'
|
||||
f'file path: {path}'
|
||||
f'parquet read took {read_delay} secs\n'
|
||||
f'polars df: {read_df}'
|
||||
)
|
||||
# write to parquet file?
|
||||
if write_parquet:
|
||||
timeframe: str = f'{secs}s'
|
||||
|
||||
if reload_parquet_to_shm:
|
||||
new = tsp.pl2np(
|
||||
deduped,
|
||||
dtype=shm.array.dtype,
|
||||
)
|
||||
# since normally readonly
|
||||
shm._array.setflags(
|
||||
write=int(1),
|
||||
)
|
||||
shm.push(
|
||||
new,
|
||||
prepend=True,
|
||||
start=new['index'][-1],
|
||||
update_first=False, # don't update ._first
|
||||
)
|
||||
datadir: Path = get_conf_dir() / 'nativedb'
|
||||
if not datadir.is_dir():
|
||||
datadir.mkdir()
|
||||
|
||||
do_markup_gaps: bool = True
|
||||
if do_markup_gaps:
|
||||
new_df: pl.DataFrame = tsp.np2pl(new)
|
||||
aids: dict = await tsp._annotate.markup_gaps(
|
||||
fqme,
|
||||
period_s,
|
||||
actl,
|
||||
new_df,
|
||||
step_gaps,
|
||||
)
|
||||
# last chance manual overwrites in REPL
|
||||
# await tractor.pause()
|
||||
if not aids:
|
||||
log.warning(
|
||||
f'No gaps were found !?\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
f'timeframe: {period_s!r}\n'
|
||||
f"WELL THAT'S GOOD NOOZ!\n"
|
||||
)
|
||||
tf2aids[period_s] = aids
|
||||
path: Path = datadir / f'{fqme}.{timeframe}.parquet'
|
||||
|
||||
else:
|
||||
# No significant gaps to handle, but may have had
|
||||
# duplicates removed (valid race conditions are ok)
|
||||
if diff > 0 and dq_issues is not None:
|
||||
log.warning(
|
||||
'Found duplicates with data quality issues '
|
||||
'but no significant time gaps!\n'
|
||||
)
|
||||
# write to fs
|
||||
start = time.time()
|
||||
df.write_parquet(path)
|
||||
delay: float = round(
|
||||
time.time() - start,
|
||||
ndigits=6,
|
||||
)
|
||||
log.info(
|
||||
f'parquet write took {delay} secs\n'
|
||||
f'file path: {path}'
|
||||
)
|
||||
|
||||
await tractor.pause()
|
||||
log.info('Exiting TSP shm anal-izer!')
|
||||
# read back from fs
|
||||
start = time.time()
|
||||
read_df: pl.DataFrame = pl.read_parquet(path)
|
||||
delay: float = round(
|
||||
time.time() - start,
|
||||
ndigits=6,
|
||||
)
|
||||
print(
|
||||
f'parquet read took {delay} secs\n'
|
||||
f'polars df: {read_df}'
|
||||
)
|
||||
|
||||
if shm_df is None:
|
||||
log.error(
|
||||
f'No matching shm buffers for {fqme} ?'
|
||||
|
||||
)
|
||||
if df is None:
|
||||
log.error(f'No matching shm buffers for {fqme} ?')
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
|
|
|||
|
|
@ -19,8 +19,7 @@
|
|||
call a poor man's tsdb).
|
||||
|
||||
AKA a `piker`-native file-system native "time series database"
|
||||
without needing an extra process and no standard TSDB features,
|
||||
YET!
|
||||
without needing an extra process and no standard TSDB features, YET!
|
||||
|
||||
'''
|
||||
# TODO: like there's soo much..
|
||||
|
|
@ -56,6 +55,8 @@ from datetime import datetime
|
|||
from pathlib import Path
|
||||
import time
|
||||
|
||||
# from bidict import bidict
|
||||
# import tractor
|
||||
import numpy as np
|
||||
import polars as pl
|
||||
from pendulum import (
|
||||
|
|
@ -63,18 +64,45 @@ from pendulum import (
|
|||
)
|
||||
|
||||
from piker import config
|
||||
from piker import tsp
|
||||
from piker.data import (
|
||||
def_iohlcv_fields,
|
||||
ShmArray,
|
||||
)
|
||||
from piker.data import def_iohlcv_fields
|
||||
from piker.data import ShmArray
|
||||
from piker.log import get_logger
|
||||
from . import TimeseriesNotFound
|
||||
|
||||
|
||||
log = get_logger('storage.nativedb')
|
||||
|
||||
|
||||
# NOTE: thanks to this SO answer for the below conversion routines
|
||||
# to go from numpy struct-arrays to polars dataframes and back:
|
||||
# https://stackoverflow.com/a/72054819
|
||||
def np2pl(array: np.ndarray) -> pl.DataFrame:
|
||||
return pl.DataFrame({
|
||||
field_name: array[field_name]
|
||||
for field_name in array.dtype.fields
|
||||
})
|
||||
|
||||
|
||||
def pl2np(
|
||||
df: pl.DataFrame,
|
||||
dtype: np.dtype,
|
||||
|
||||
) -> np.ndarray:
|
||||
|
||||
# Create numpy struct array of the correct size and dtype
|
||||
# and loop through df columns to fill in array fields.
|
||||
array = np.empty(
|
||||
df.height,
|
||||
dtype,
|
||||
)
|
||||
for field, col in zip(
|
||||
dtype.fields,
|
||||
df.columns,
|
||||
):
|
||||
array[field] = df.get_column(col).to_numpy()
|
||||
|
||||
return array
|
||||
|
||||
|
||||
def detect_period(shm: ShmArray) -> float:
|
||||
'''
|
||||
Attempt to detect the series time step sampling period
|
||||
|
|
@ -95,19 +123,16 @@ def detect_period(shm: ShmArray) -> float:
|
|||
|
||||
def mk_ohlcv_shm_keyed_filepath(
|
||||
fqme: str,
|
||||
period: float | int, # ow known as the "timeframe"
|
||||
period: float, # ow known as the "timeframe"
|
||||
datadir: Path,
|
||||
|
||||
) -> Path:
|
||||
) -> str:
|
||||
|
||||
if period < 1.:
|
||||
raise ValueError('Sample period should be >= 1.!?')
|
||||
|
||||
path: Path = (
|
||||
datadir
|
||||
/
|
||||
f'{fqme}.ohlcv{int(period)}s.parquet'
|
||||
)
|
||||
period_s: str = f'{period}s'
|
||||
path: Path = datadir / f'{fqme}.ohlcv{period_s}.parquet'
|
||||
return path
|
||||
|
||||
|
||||
|
|
@ -161,13 +186,7 @@ class NativeStorageClient:
|
|||
|
||||
def index_files(self):
|
||||
for path in self._datadir.iterdir():
|
||||
if (
|
||||
path.is_dir()
|
||||
or
|
||||
'.parquet' not in str(path)
|
||||
# or
|
||||
# path.name in {'borked', 'expired',}
|
||||
):
|
||||
if path.name in {'borked', 'expired',}:
|
||||
continue
|
||||
|
||||
key: str = path.name.rstrip('.parquet')
|
||||
|
|
@ -209,21 +228,8 @@ class NativeStorageClient:
|
|||
fqme,
|
||||
timeframe,
|
||||
)
|
||||
except FileNotFoundError as fnfe:
|
||||
|
||||
bs_fqme, _, *_ = fqme.rpartition('.')
|
||||
|
||||
possible_matches: list[str] = []
|
||||
for tskey in self._index:
|
||||
if bs_fqme in tskey:
|
||||
possible_matches.append(tskey)
|
||||
|
||||
match_str: str = '\n'.join(sorted(possible_matches))
|
||||
raise TimeseriesNotFound(
|
||||
f'No entry for `{fqme}`?\n'
|
||||
f'Maybe you need a more specific fqme-key like:\n\n'
|
||||
f'{match_str}'
|
||||
) from fnfe
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
|
||||
times = array['time']
|
||||
return (
|
||||
|
|
@ -236,7 +242,6 @@ class NativeStorageClient:
|
|||
self,
|
||||
fqme: str,
|
||||
period: float,
|
||||
|
||||
) -> Path:
|
||||
return mk_ohlcv_shm_keyed_filepath(
|
||||
fqme=fqme,
|
||||
|
|
@ -244,23 +249,6 @@ class NativeStorageClient:
|
|||
datadir=self._datadir,
|
||||
)
|
||||
|
||||
def _cache_df(
|
||||
self,
|
||||
fqme: str,
|
||||
df: pl.DataFrame,
|
||||
timeframe: float,
|
||||
|
||||
) -> None:
|
||||
# cache df for later usage since we (currently) need to
|
||||
# convert to np.ndarrays to push to our `ShmArray` rt
|
||||
# buffers subsys but later we may operate entirely on
|
||||
# pyarrow arrays/buffers so keeping the dfs around for
|
||||
# a variety of purposes is handy.
|
||||
self._dfs.setdefault(
|
||||
timeframe,
|
||||
{},
|
||||
)[fqme] = df
|
||||
|
||||
async def read_ohlcv(
|
||||
self,
|
||||
fqme: str,
|
||||
|
|
@ -269,20 +257,13 @@ class NativeStorageClient:
|
|||
# limit: int = int(200e3),
|
||||
|
||||
) -> np.ndarray:
|
||||
path: Path = self.mk_path(
|
||||
fqme,
|
||||
period=int(timeframe),
|
||||
)
|
||||
path: Path = self.mk_path(fqme, period=int(timeframe))
|
||||
df: pl.DataFrame = pl.read_parquet(path)
|
||||
self._dfs.setdefault(timeframe, {})[fqme] = df
|
||||
|
||||
self._cache_df(
|
||||
fqme=fqme,
|
||||
df=df,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
# TODO: filter by end and limit inputs
|
||||
# times: pl.Series = df['time']
|
||||
array: np.ndarray = tsp.pl2np(
|
||||
array: np.ndarray = pl2np(
|
||||
df,
|
||||
dtype=np.dtype(def_iohlcv_fields),
|
||||
)
|
||||
|
|
@ -292,15 +273,11 @@ class NativeStorageClient:
|
|||
self,
|
||||
fqme: str,
|
||||
period: int = 60,
|
||||
load_from_offline: bool = True,
|
||||
|
||||
) -> pl.DataFrame:
|
||||
try:
|
||||
return self._dfs[period][fqme]
|
||||
except KeyError:
|
||||
if not load_from_offline:
|
||||
raise
|
||||
|
||||
await self.read_ohlcv(fqme, period)
|
||||
return self._dfs[period][fqme]
|
||||
|
||||
|
|
@ -322,39 +299,32 @@ class NativeStorageClient:
|
|||
datadir=self._datadir,
|
||||
)
|
||||
if isinstance(ohlcv, np.ndarray):
|
||||
df: pl.DataFrame = tsp.np2pl(ohlcv)
|
||||
df: pl.DataFrame = np2pl(ohlcv)
|
||||
else:
|
||||
df = ohlcv
|
||||
|
||||
self._cache_df(
|
||||
fqme=fqme,
|
||||
df=df,
|
||||
timeframe=timeframe,
|
||||
)
|
||||
|
||||
# TODO: in terms of managing the ultra long term data
|
||||
# -[ ] use a proper profiler to measure all this IO and
|
||||
# - use a proper profiler to measure all this IO and
|
||||
# roundtripping!
|
||||
# -[ ] implement parquet append!? see issue:
|
||||
# https://github.com/pikers/piker/issues/536
|
||||
# -[ ] try out ``fastparquet``'s append writing:
|
||||
# https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write
|
||||
# - try out ``fastparquet``'s append writing:
|
||||
# https://fastparquet.readthedocs.io/en/latest/api.html#fastparquet.write
|
||||
start = time.time()
|
||||
df.write_parquet(path)
|
||||
delay: float = round(
|
||||
time.time() - start,
|
||||
ndigits=6,
|
||||
)
|
||||
log.info(
|
||||
print(
|
||||
f'parquet write took {delay} secs\n'
|
||||
f'file path: {path}'
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
async def write_ohlcv(
|
||||
self,
|
||||
fqme: str,
|
||||
ohlcv: np.ndarray | pl.DataFrame,
|
||||
ohlcv: np.ndarray,
|
||||
timeframe: int,
|
||||
|
||||
) -> Path:
|
||||
|
|
@ -406,8 +376,6 @@ class NativeStorageClient:
|
|||
# ...
|
||||
|
||||
|
||||
# TODO: does this need to be async on average?
|
||||
# I guess for any IPC connected backend yes?
|
||||
@acm
|
||||
async def get_client(
|
||||
|
||||
|
|
@ -425,7 +393,7 @@ async def get_client(
|
|||
'''
|
||||
datadir: Path = config.get_conf_dir() / 'nativedb'
|
||||
if not datadir.is_dir():
|
||||
log.info(f'Creating `nativedb` dir: {datadir}')
|
||||
log.info(f'Creating `nativedb` director: {datadir}')
|
||||
datadir.mkdir()
|
||||
|
||||
client = NativeStorageClient(datadir)
|
||||
|
|
|
|||
|
|
@ -18,12 +18,24 @@
|
|||
Toolz for debug, profile and trace of the distributed runtime :surfer:
|
||||
|
||||
'''
|
||||
from tractor.devx import (
|
||||
open_crash_handler as open_crash_handler,
|
||||
from .debug import (
|
||||
open_crash_handler,
|
||||
)
|
||||
from .profile import (
|
||||
Profiler as Profiler,
|
||||
pg_profile_enabled as pg_profile_enabled,
|
||||
ms_slower_then as ms_slower_then,
|
||||
timeit as timeit,
|
||||
Profiler,
|
||||
pg_profile_enabled,
|
||||
ms_slower_then,
|
||||
timeit,
|
||||
)
|
||||
|
||||
# TODO: other mods to include?
|
||||
# - DROP .trionics, already moved into tractor
|
||||
# - move in `piker.calc`
|
||||
|
||||
__all__: list[str] = [
|
||||
'open_crash_handler',
|
||||
'pg_profile_enabled',
|
||||
'ms_slower_then',
|
||||
'Profiler',
|
||||
'timeit',
|
||||
]
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue