I have a subscriber which pushes data into queues. Now the messages looks this

{
 "Content": {
   "_id" ""5ceya67bbsbag3",
   "dataset": { 
     "upper": {},
      "lower": {}

}
}

Now a new message can be pushed with same content id but data will be different. So in that i want to delete the old message with same id or replaece the message those id is same & retain only latest message.

I have not found a direct solution for this in rabbitmq. Please guide me how we can do this ?

I have already gone through some posts.

Post 1

Post 2

I have a subscriber which pushes data into queues. Now the messages looks this

{
 "Content": {
   "_id" ""5ceya67bbsbag3",
   "dataset": { 
     "upper": {},
      "lower": {}

}
}

Now a new message can be pushed with same content id but data will be different. So in that i want to delete the old message with same id or replaece the message those id is same & retain only latest message.

I have not found a direct solution for this in rabbitmq. Please guide me how we can do this ?

I have already gone through some posts.

Post 1

Post 2

• javascript
• node.js
• rabbitmq
• message-queue
• amqp

Share Improve this question Follow asked Jun 26, 2020 at 7:04 TechChainTechChain 8,9603232 gold badges117117 silver badges244244 bronze badges 1

• 1 so you have msg1 with content id M. You start processing it and in the middle of processing msg2 with content id M gets into queue. What do you do in that case? Answer to that question is also answer to your question. So what you want RMQ to do - the consumers should be doing. – cantSleepNow Commented Jul 1, 2020 at 18:45

Add a ment  | 

3 Answers 3

Sorted by: Reset to default 5 +25

What you are trying to achieve cannot be trivially solved with RabbitMQ (or rather the AMQP protocol).

RabbitMQ queues are simple FIFO stacks and don't offer any mean of access to the elements beyond publishing at their top and consuming from their bottom.

Therefore, the only way to "update" an already existing message without relying on an another service would be to fetch all the messages until you find the one you are interested in, discard it, and publish the new one with the other messages you fetched together with it.

Overall, the remendation when using RabbitMQ in regards of message duplication is to make their consumption idempotent. In other words, the consumption of 2 messages deemed to be the same should lead to the same oute.

One way to achieve idempotency is to rely on a secondary cache where you store the message identifiers and their validity. Once a consumer fetches a new message from RabbitMQ, it would check the cache to see if it's a valid message or not and act accordingly.

I think this is a slightly wrong way to use rabbitMQ.

only immutable (not intended to change) tasks should be put into queues which a worker shall consume.

An alternative way to implement your particular task is

• just push immutable data into queue { "content" : { "_id" : "5ceya67bbsbag3"} .. }
• store mutable data in db (mongo) or in-mem db (something like redis is suggested here).
• whenever update needed, update in db
• let your worker fetch required data using your "_id" ref from the db

I am not sure if removing a message is a good idea. If your requirement is to update the data as it es so that always latest data is maintained for same Id.

Other thing is as messages are getting consumed always the last message data will get updated. So I don't see a issue here in Rabbit MQ.