We have long-lived connections that recreates itself when broken:

class RMQ():
    def __init__(self):
        self.connection = ...

    def recreate(self):
        self.connection = ...

    def publish(self):
        """All methods have retry logic to restore connection."""
        try:
            self.connection.channel.basic_publish(...)
        except AMQPConnectionError:
            self.recreate()
            self.connection.channel.basic_publish(...)

    def __del__(self):
        if self.connection.is_open:
            self.connection.close()

Should we explicitly close such connections during program shutdown or will Pika/Python/OS take care of such things?

If we have to do it, where should we do it: atexit or some other place? Context managers does not seem applicable during shutdown as connections live forever (decision can't be taken during each call).

Putting this logic in __del__ doesn't seem to work as the underlying socket is already closed(!) (that is the exception thrown with SSL connections). Pytest hangs if we simulate a reconnection (deadlock). __del__ behaviour is unpredictable as per docs, PyMySQL issue 961 and Pythonic way to close connections.

We have long-lived connections that recreates itself when broken:

class RMQ():
    def __init__(self):
        self.connection = ...

    def recreate(self):
        self.connection = ...

    def publish(self):
        """All methods have retry logic to restore connection."""
        try:
            self.connection.channel.basic_publish(...)
        except AMQPConnectionError:
            self.recreate()
            self.connection.channel.basic_publish(...)

    def __del__(self):
        if self.connection.is_open:
            self.connection.close()

Should we explicitly close such connections during program shutdown or will Pika/Python/OS take care of such things?

If we have to do it, where should we do it: atexit or some other place? Context managers does not seem applicable during shutdown as connections live forever (decision can't be taken during each call).

Putting this logic in __del__ doesn't seem to work as the underlying socket is already closed(!) (that is the exception thrown with SSL connections). Pytest hangs if we simulate a reconnection (deadlock). __del__ behaviour is unpredictable as per docs, PyMySQL issue 961 and Pythonic way to close connections.

• python
• rabbitmq
• pika

Share Improve this question Follow edited Mar 10 at 16:27 Nishant asked Mar 10 at 13:23 NishantNishant 22k2020 gold badges7878 silver badges106106 bronze badges 3

• 2 Context managers are still applicable; the idea is that an instance of RMQ receives an open connection, rather than opening one itself, and the context manager delimits the useful lifetime of the RMQ object. – chepner Commented Mar 10 at 13:27
• That is, separate the creation and destruction of the connection from the process that uses the connection. – chepner Commented Mar 10 at 13:27
• Thanks @chepner. I think SQLAlchemy has something called pool_pre_ping to detect closed connections and its session manager could be doing something similar. However, I think, Pika's doesn't have similar OOB, and it can't detect a closed connection without performing an action: github/pika/pika/issues/877#issuecomment-366430873. Maybe an __exit__ handler might be worth checking. This problem is like: what to do when program shuts down. – Nishant Commented Mar 10 at 14:32

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Using atexit or signal handlers is the best way to ensure a controlled and clean shutdown. __del__ should be avoided due to unpredictable behavior.

import signal
import sys

class RMQ:
    def __init__(self):
        self.connection = ...  # Initialize connection
        signal.signal(signal.SIGTERM, self.close)
        signal.signal(signal.SIGINT, self.close)

    def close(self, *args):
        if self.connection and self.connection.is_open:
            self.connection.close()
        sys.exit(0)

rmq = RMQ()