From 21baee4da9c88b55fc11a9edb977fd63289d74d8 Mon Sep 17 00:00:00 2001
From: Tyler Goodlet <jgbt@protonmail.com>
Date: Sun, 26 Feb 2023 16:50:12 -0500
Subject: [PATCH] Update for latest history ep and more thorough high-level
 overview

---
 piker/broker/README.rst | 95 ++++++++++++++++++++++++++++++++---------
 1 file changed, 76 insertions(+), 19 deletions(-)

diff --git a/piker/broker/README.rst b/piker/broker/README.rst
index 5a778de8..2b7866a3 100644
--- a/piker/broker/README.rst
+++ b/piker/broker/README.rst
@@ -3,15 +3,31 @@ provider "spec" (aka backends)
 ``piker`` abstracts and encapsulates real-time data feeds across a slew
 of providers covering many (pretty much any) instrument class.
 
-
-this is shoddy attempt as specifying what a backend must provide
-as a basic api, per functionality-feature set, in order to be
-supported in zee sytem.
+This doc is shoddy attempt as specifying what a backend must provide as
+a basic api, per functionality-feature set, in order to be supported for
+bny of real-time and historical data feeds and order control via the
+``emsd`` clearing system.
 
 "providers" must offer a top plevel namespace (normally exposed as
 a python module) which offers to a certain set of (async) functions
 to deliver info through a real-time, normalized data layer.
 
+Generally speaking we break each ``piker.brokers.<backend_name>`` into
+a python package containing 3 sub-modules:
+- ``.api`` containing lowest level client code used to interact
+  specifically with the APIs of the exchange, broker or data provider.
+- ``.feed`` which provides historical and real-time quote stream data
+  provider endpoints called by piker's data layer in
+  ``piker.data.feed``.
+- ``.broker`` which defines endpoints expected by
+  ``pikerd.clearing._ems`` and which are expected to adhere to the msg
+  protocol defined in ``piker.clrearing._messages``.
+
+
+Our current set of "production" grade backends includes:
+- ``kraken``
+- ``ib``
+
 
 data feeds
 ----------
@@ -63,28 +79,69 @@ further streamed quote messages should be in this same format.
 
 historical OHLCV sampling
 -------------------------
+Example endpoint copyed from the ``binance`` backend:
 
 .. code:: python
 
-    async def backfill_bars(
-        sym: str,
-        shm: ShmArray,  # type: ignore # noqa
+   @acm
+    async def open_history_client(
+        symbol: str,
 
-        count: int = 10,  # NOTE: any more and we'll overrun the underlying buffer
+    ) -> tuple[Callable, int]:
 
-        # startup sync via ``trio``
-        task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
+        # TODO implement history getter for the new storage layer.
+        async with open_cached_client('binance') as client:
 
-    ) -> None:
+            async def get_ohlc(
+                timeframe: float,
+                end_dt: datetime | None = None,
+                start_dt: datetime | None = None,
+
+            ) -> tuple[
+                np.ndarray,
+                datetime,  # start
+                datetime,  # end
+            ]:
+                if timeframe != 60:
+                    raise DataUnavailable('Only 1m bars are supported')
+
+                array = await client.bars(
+                    symbol,
+                    start_dt=start_dt,
+                    end_dt=end_dt,
+                )
+                times = array['time']
+                if (
+                    end_dt is None
+                ):
+                    inow = round(time.time())
+                    if (inow - times[-1]) > 60:
+                        await tractor.breakpoint()
+
+                start_dt = pendulum.from_timestamp(times[0])
+                end_dt = pendulum.from_timestamp(times[-1])
+
+                return array, start_dt, end_dt
+
+            yield get_ohlc, {'erlangs': 3, 'rate': 3}
+
+
+This `@acm` routine is responsible for setting up an async historical
+data query routine for both charting and any local storage requirements.
+
+The returned async func should retreive, normalize and deliver
+a ``tuple[np.ndarray, pendulum.dateime, pendulum.dateime]`` of the the
+``numpy``-ified data, the start and stop datetimes for the delivered
+history "frame". The history backloading routines inside
+``piker.data.feed`` expect this interface for both loading history into
+``ShmArrayt`` real-time buffers as well as any configured
+time-series-database (tsdb) and  normally the format of this data is
+OHLCV sampled price and volume data but in theory can be high
+reslolution tick/trades/book times series in the future.
+
+Currently sampling routines for charting and fsp processing expects
+a max resolution of 1s (second) OHLCV sampled data.
 
-this routine is responsible for setting up historical data for both
-charting and any local storage requirements. it should retreive, normalize and
-push the data to shared memory. normally the format of this data is
-OHLCV sampled price and volume data but can be high reslolution
-tick/trades/book times series as well. currently charting can expects
-a max resolution of 1s (second) OHLCV sampled data (any more then this
-and you probably can't do much with it as a manual trader; this is
-obviously not true of automatic strats realized as code).
 
 OHLCV minmal schema
 ********************