I am observing my Galera Cluster with Prometheus.

Galera has a metric that gives the percentage of time the cluster's nodes spent paused due to flow control since the last FLUSH STATUS.

Luckily, there's a metric that gives the uptime in seconds since the last FLUSH STATUS.

The problem is that either metric is not super helpful on its own.

This only tells me that the percentage of time spent paused is going down. A rate on this tells me the same thing, because the percentage decreases with uptime.

So, I'd have to FLUSH STATUS on the regular to see how the cluster is behaving in a specific instant, which is annoying.

Since these metrics are both relative to the same point in time, I can multiply them to get a useful metric that is constant when there are no changes, rather than reducing at a steady rate.

"Perfect", I thought, "I'll take the rate to get a useful, reactive metric."

rate((mysql_global_status_wsrep_flow_control_paused * mysql_global_status_uptime_since_flush_status)[$__rate_interval])

Well. Since I got that message, I've been trying to Google and GPT my way out of this, but I can't seem to grok it, and neither does Claude. Clearly, I'm hitting a wall in my PromQL / Grafana fundamentals, so I must pray at the altar of Stackoverflow.

May you Grafanic Oracles please spare me your wisdom?

I am observing my Galera Cluster with Prometheus.

Galera has a metric that gives the percentage of time the cluster's nodes spent paused due to flow control since the last FLUSH STATUS.

Luckily, there's a metric that gives the uptime in seconds since the last FLUSH STATUS.

The problem is that either metric is not super helpful on its own.

This only tells me that the percentage of time spent paused is going down. A rate on this tells me the same thing, because the percentage decreases with uptime.

So, I'd have to FLUSH STATUS on the regular to see how the cluster is behaving in a specific instant, which is annoying.

Since these metrics are both relative to the same point in time, I can multiply them to get a useful metric that is constant when there are no changes, rather than reducing at a steady rate.

"Perfect", I thought, "I'll take the rate to get a useful, reactive metric."

rate((mysql_global_status_wsrep_flow_control_paused * mysql_global_status_uptime_since_flush_status)[$__rate_interval])

Well. Since I got that message, I've been trying to Google and GPT my way out of this, but I can't seem to grok it, and neither does Claude. Clearly, I'm hitting a wall in my PromQL / Grafana fundamentals, so I must pray at the altar of Stackoverflow.

May you Grafanic Oracles please spare me your wisdom?

• prometheus
• grafana

Share Improve this question Follow asked Mar 13 at 3:43 ProudOneProudOne 38944 silver badges1717 bronze badges

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 2

Note that the rate function must be applied to counters only. If it is applied to other metric types, then it may return arbitrary garbage. So, in your case the deriv function must be used instead of rate. Also, the rollup functions expect time range series selector as their input. In your case you pass the result of multiplication to the rate function instead of series selector. This could work, but you need to use subqueries . E.g. you need to add :step suffix inside square brackets. Given the above details the following query could return the expected results:

deriv((mysql_global_status_wsrep_flow_control_paused * mysql_global_status_uptime_since_flush_status)[1m:10s])

I didn't see the trees for the forest on this one. There already is a combined metric, no multiplication necessary.

wsrep_flow_control_paused_ns