I’d like your advice on the design of my application.
I use websockets to receive new data and the request module to retrieve older data.
Then I use pyqtgraph to display data and tables etc with pyqt5.
There are some data that I don’t keep in memory, I just display them on screen without the possibility to interact with them, and I have other data that I keep in memory, with which I do some processing.
I would like to know if I should use dictionaries to store and process data or create a database with SQL or use pandas.
There will be a lot of inserting, extracting, deleting and a lot of calculations.
Potentially, when there are big moves, I could have thousands of messages per second to process, which I would have to add to my database, process and then display them on screen or do whatever I wanted with them.
If you have any questions, don’t hesitate.
Example of connection:
import websockets import asyncio import json async def capture_data(): subscriptions = [sub for sub in ["quote", "trade", "instrument"]] uri = "wss://www.bitmex.com/realtime?subscribe=" + ",".join(subscriptions) async with websockets.connect(uri) as websocket: while True: data = await websocket.recv() print(json.loads(data)) asyncio.get_event_loop().run_until_complete(capture_data())
Go to Source