I'm trying to debug a problem in our production Kafka Streams app. The (simplified) topology looks something like this

builder.stream("input").groupByKey().reduce(
  (agg, val) -> "bar",
  Materialized.as("saved_store")
);

99% of our incoming messages on the "input" topic seem to work fine, but I found some cases that didn't. I see a single message for a particular key 123 in "input"

"123": "foo"

but when I look at the private Kafka topic that "saved_store" is materializing to, there are two messages there and both of them look like

"123": "bar"

What could be causing this? I would expect the saved store to only have a single message AND for value to be "foo", since the reducer is not supposed to be called when the key is seen for the first time.

I'm trying to debug a problem in our production Kafka Streams app. The (simplified) topology looks something like this

builder.stream("input").groupByKey().reduce(
  (agg, val) -> "bar",
  Materialized.as("saved_store")
);

99% of our incoming messages on the "input" topic seem to work fine, but I found some cases that didn't. I see a single message for a particular key 123 in "input"

"123": "foo"

but when I look at the private Kafka topic that "saved_store" is materializing to, there are two messages there and both of them look like

"123": "bar"

What could be causing this? I would expect the saved store to only have a single message AND for value to be "foo", since the reducer is not supposed to be called when the key is seen for the first time.

• apache-kafka
• apache-kafka-streams
• reduce

Share Improve this question Follow edited Nov 20, 2024 at 20:54 Rob 15.2k3030 gold badges4848 silver badges7373 bronze badges asked Nov 20, 2024 at 20:32 EgorEgor 1,6601313 silver badges2626 bronze badges 1

• Did you try disable caching to remove one moving piece? – Matthias J. Sax Commented Dec 29, 2024 at 18:11

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

look at the private Kafka topic that "saved_store" is materializing to,

That's the wrong way to view a store. Compacted topics back stores. Compacted topics do not guarantee unique keys within an open segment, before a log cleaner thread runs

"Uncompacted' topics Vs compacted topics