I have two services calling back to each other multiple times via httpx.Client():

1. api_service
2. queue_service
3. [all other services]

calls and callbacks between 1. and 3. work fine.

1. calls 2. where 2. callsback to 1. for other (dynamic) information. this is where 2. 500's with httpx.RemoteProtocolError: Server disconnected without sending a response.

I am unable to recreate this on my local machine as it works fine, so it leaves me to believe the issue is related to the system settings or hardware on my production machine:

OS Details

Distributor ID: Ubuntu

Description: Ubuntu 22.04.5 LTS

Release: 22.04

Codename: jammy

Machine Details

description: Rack Mount Chassis

product: ThinkSystem SR650 -[7X06CTO1WW]- (7X06CTO1WW)

vendor: Lenovo

This is the code snippet that seems to be receiving the RemoteProtocolError:

async def post(self, ctx, url, json_data: dict = {}, success_status_code: int = 200, raise_on_failure=True):

    url = self.service_url + url
    async with httpx.AsyncClient(timeout=None) as client:
        content = json.stringify(json_data) if json_data else None
        resp = await client.post(url, content=content,
                                 headers={**self._get_session_header(ctx), **{"Content-Type": "application/json", "Connection": "close"}})

    return self._handle_return(resp, success_status_code, raise_on_failure=raise_on_failure)

This code snippet is the exact same as it is in 1. api_service and 3. [all other services]. I've gone as far as migrating the Class to using aiohttp but the error returns with similar error aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected.

Here is the aiohttp code snippet:

async def post(self, ctx, url, json_data: dict = {}, success_status_code: int = 200, raise_on_failure=True):

    url = self.service_url + url
    async with aiohttp.ClientSession(connector = aiohttp.TCPConnector(limit=50), timeout = aiohttp.ClientTimeout(total=None, sock_connect=None, sock_read=None)) as client:
        content = json.stringify(json_data) if json_data else None
        resp = await client.post(url, data=content,
                                 headers={**self._get_session_header(ctx), **{"Content-Type": "application/json", "Connection": "close"}})

        return await self._handle_return(resp, success_status_code, raise_on_failure=raise_on_failure)

I've also increased the uvicorn keep-alive-timeout 30 and uvicorn --workers 4 and this returns the same result for both. This is the current supervisord command:

command = uvicorn app.api:app --host 0.0.0.0 --port 80 --reload --timeout-keep-alive 30 --workers 4

I have this running on a docker-compose.yml file. Thanks.