I’d like your advice on the design of my application.
I use websockets to receive new data and the request module to retrieve older data.
Then I use pyqtgraph to display data and tables etc with pyqt5.
There are some data that I don’t keep in memory, I just display them on screen without the possibility to interact with them, and I have other data that I keep in memory, with which I do some processing.
I would like to know if I should use dictionaries to store and process data or create a database with SQL or use pandas.
There will be a lot of inserting, extracting, deleting and a lot of calculations.
Potentially, when there are big moves, I could have thousands of messages per second to process, which I would have to add to my database, process and then display them on screen or do whatever I wanted with them.
If you have any questions, don’t hesitate.
Example of connection:
import websockets
import asyncio
import json
async def capture_data():
subscriptions = [sub for sub in ["quote", "trade", "instrument"]]
uri = "wss://www.bitmex.com/realtime?subscribe=" + ",".join(subscriptions)
async with websockets.connect(uri) as websocket:
while True:
data = await websocket.recv()
print(json.loads(data))
asyncio.get_event_loop().run_until_complete(capture_data())
Go to Source
Author: antho