I'm experiencing slow performance when parsing long JSON strings retrieved from Redis in Python. The process involves fetching a large string (a list of dicts with same structure) stored in Redis, then deserializing it into json for further processing. Some lists contains over 100000 items and have size more then 70mb. For such lists i observe time, consumed for converting string to json around 1.5-4 seconds. I tried ujson, orjson, msgspec, msgpack, simdjson, redis-json, but all are giving approximately same result. The best timing of 1.5s i got with msgspec.json.Decoder and providing struct for data, but it is still more then i need. What can be another options, assuming that i have list of dicts with over 100000 items and size over 70mb, which i need to store, and then it should be used in apps from different containers (including highly loaded fastApi service). There are over 10k such lists, but the most of them are small and there are no issues with them, only like 10-15% are big enough to slower overall performance. Any ideas are appreciated, even completely different approach for solving this task.Structure of stored data:

to_save = List[Route]
    
class Route(msgspec.Struct):
    poos: List[str]
    asts: List[str]
    has_limit: str
    items: List[Item]
    is_v5: bool
    datas: int
    actions: int
    
class Item(msgspec.Struct):
    fro: int
    to: int
    header: int
    type: str
    actions: int
    datas: int
    params: str

There was attempt to solve it using graphs, so here is brief description of network:

Network description:

1. There are Nodes (N_1..N_300) and subNodes (sn_1..sn_400)
2. Each node contains from 2 to 14 subnodes with unique labels, different nodes can contain subnodes with same label, but different attribute's values
3. Task is to find best path from any subnode_X to any subnode_Y
4. Network is dynamic, nodes and subnodes can add/remove, values of their attributes also are changing. Current setup (worked well until requests count increased significally):
5. Request is coming with START subnode label, END subnode label and list of input parameters
6. All valid paths are loaded from Redis or discovered and stored as one entry (slow if many paths)
7. Actual data for all nodes and subnodes is loaded from Redis and used to calculate each route to determine best (fast enough currently)
8. If any node/subnode is added/removed all stored paths are cleared

First attempt to move to graph based DB (Memgraph) failed due to zero experience in graphs, restrictions for some paths, quite complex calculations and fact, that current worst path can quickly become best due to different input parameters and involved nodes/subnodes state change. I understand that there are very low chances to get direct answer for such question, need at least pointing to proper direction.