09 - Monitoring Kafka: JMX Exporter, Prometheus and Grafana Dashboard

How you know the cluster is unwell before your users tell you

A slow broker does not produce an error: it produces a delay. A partition with a lagging replica sets off nothing until the leader goes down too. A consumer group that cannot keep up does not stop: it accumulates lag, and after a few hours the data it reads is a day old. Kafka monitoring exists to turn these silent failures into numbers with a threshold and a person who receives the alert. The goal of this article is modest and verifiable: to know which few metrics deserve an alarm, where they are read from, and how the rule is written.

A note on method first. Here there is no Kafka cluster and no Prometheus: no dashboard was seen. Instead I ran two Java 21 models (a fake exporter that applies rules to real JVM MBeans, and a classifier of the lag trend) and checked the YAML syntax of the configuration files with a parser. The metric names come from the Kafka 4.4 "Monitoring" documentation, the exporter rules from the JMX Exporter 1.7.0 documentation and the alerts from the Prometheus one, consulted on 2026-11-12. The example figures in the listings are made up for the test and I say so where they appear.

The few metrics that have a threshold

The Kafka documentation lists hundreds of metrics. For an alarm a few are enough, because they have an expected value that does not depend on your load:

• kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions: partitions where the ISR is smaller than the set of replicas. The expected value is 0.
• kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount: partitions where the ISR is below min.insync.replicas. Expected 0.
• kafka.controller:type=KafkaController,name=OfflinePartitionsCount: partitions without a leader, as seen by that controller.
• kafka.controller:type=KafkaController,name=ActiveControllerCount: the documentation says that across the whole cluster only one node should have 1.
• The lag of consumer groups, which is not a broker metric (we see it further down).

The criterion that holds them together is the question "how close am I to losing data or to not serving it?". The first two measure the loss of durability margin (the second is already an operational problem: with acks=all the writes to those partitions fail, as seen in the article on advanced producers). The third measures not serving. The fourth checks that the cluster has a single brain, and it was born with KRaft and its controller quorum. All of them have a right value, 0 or 1, so the threshold requires no experience of the load: that is where they are most useful. The rest (request latency, bytes in, internal queues) is context to show on a dashboard and not to alarm on until you have a historical reference.

How they leave the broker: JMX, and an exporter that translates them

Kafka exposes metrics as JMX MBeans, each with a name in the form domain:type=...,name=.... The documentation specifies that remote JMX access is off by default: it is turned on with the JMX_PORT variable or with the standard JVM properties, and it must be protected in production. Prometheus does not speak JMX, so a translator is needed: the JMX Exporter, which by its own documentation is usually used as a Java agent inside the broker process, avoiding the configuration of remote JMX.

The exporter works by rules. Each rule has a pattern (a regular expression), a name (the Prometheus metric name), and optionally labels, value and type. The documentation describes the input to which the pattern applies: domain<property1=value1, property2=value2><keys>attribute: value. The rules are evaluated in order, the first one that matches wins, and attributes without a rule are not collected. The listing reproduces this mechanism on real JVM MBeans: it registers four MBeans with names identical to those of Kafka (the values are made up), builds the input string for each and applies three rules.

import javax.management.*;
import java.lang.management.ManagementFactory;
import java.util.*;
import java.util.regex.*;

public class RegoleJmx {
    public interface ValoreMBean { long getValue(); }               // an MBean with the "Value" attribute
    public static class Valore implements ValoreMBean {
        final long v; Valore(long v) { this.v = v; }
        public long getValue() { return v; }
    }

    // A rule as in the exporter documentation: pattern, name, labels (with capture groups).
    record Regola(Pattern pattern, String name, Map<String, String> labels) {}

    public static void main(String[] args) throws Exception {
        MBeanServer server = ManagementFactory.getPlatformMBeanServer();
        // MBeans with the same names as those documented by Kafka; the values are made up for the test.
        server.registerMBean(new Valore(0), new ObjectName("kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions"));
        server.registerMBean(new Valore(1), new ObjectName("kafka.controller:type=KafkaController,name=ActiveControllerCount"));
        server.registerMBean(new Valore(12), new ObjectName("kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=orders"));
        server.registerMBean(new Valore(7), new ObjectName("kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=click"));

        List<Regola> regole = List.of(
            new Regola(Pattern.compile("kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value: (.*)"),
                "kafka_server_replicamanager_underreplicatedpartitions", Map.of()),
            new Regola(Pattern.compile("kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value: (.*)"),
                "kafka_controller_activecontrollercount", Map.of()),
            new Regola(Pattern.compile("kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>Value: (.*)"),
                "kafka_server_messages_in", Map.of("topic", "$1")));

        for (ObjectName on : new TreeSet<>(server.queryNames(new ObjectName("kafka.*:*"), null))) {
            // The documented form of the input: domain<properties><>attribute: value
            StringBuilder props = new StringBuilder();
            for (String p : on.getKeyPropertyListString().split(",")) props.append(props.length() == 0 ? "" : ", ").append(p);
            String ingresso = on.getDomain() + "<" + props + "><>Value: " + server.getAttribute(on, "Value");
            for (Regola r : regole) {
                Matcher m = r.pattern().matcher(ingresso);
                if (!m.matches()) continue;
                StringBuilder riga = new StringBuilder(r.name());
                if (!r.labels().isEmpty()) {
                    riga.append("{");
                    r.labels().forEach((k, v) -> riga.append(k).append("=\"").append(v.replace("$1", m.group(1))).append("\""));
                    riga.append("}");
                }
                System.out.println(riga + " " + m.group(m.groupCount()));
                break;                                               // the first matching rule wins
            }
        }
    }
}

// Output:
// kafka_controller_activecontrollercount 1
// kafka_server_messages_in{topic="click"} 7
// kafka_server_messages_in{topic="orders"} 12
// kafka_server_replicamanager_underreplicatedpartitions 0

The result is a small Prometheus-style output: the number of active controllers, the under-replicated partitions and the rate of incoming messages with the topic label extracted from the capture group. The two kafka_server_messages_in lines have different labels because the pattern captures the topic name, and the value is that of the group after it. The three things to take away are in the code. The pattern ends with Value: (.*) because the input also contains the value. A property with a variable name, like topic=(.+), becomes a label and not a distinct metric. And the break after the first match replicates the "first one wins" rule. The model is not the exporter: it has no cache, does not handle composite data and does not know the types. That is why the real configuration you find below is a template, validated in syntax and not run.

# Template to adapt (not run): exporter rules in the format of the JMX Exporter 1.7.0 documentation.
# The rules are evaluated in order and it stops at the first one that matches; attributes without a rule are not collected.
rules:
  - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value: (.*)'
    name: kafka_server_replicamanager_underreplicatedpartitions
    value: '$1'
    type: GAUGE
  - pattern: 'kafka.server<type=ReplicaManager, name=UnderMinIsrPartitionCount><>Value: (.*)'
    name: kafka_server_replicamanager_underminisrpartitioncount
    value: '$1'
    type: GAUGE
  - pattern: 'kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value: (.*)'
    name: kafka_controller_offlinepartitionscount
    value: '$1'
    type: GAUGE
  - pattern: 'kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value: (.*)'
    name: kafka_controller_activecontrollercount
    value: '$1'
    type: GAUGE
  - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>OneMinuteRate: (.*)'
    name: kafka_server_messagesinpersec_oneminuterate
    value: '$2'
    labels:
      topic: '$1'
    type: GAUGE

It is activated by adding the Java agent to the broker, for example with KAFKA_OPTS="-javaagent:/opt/kafka/jmx_prometheus_javaagent-1.7.0.jar=9404:/opt/kafka/exporter.yml", following the format of the exporter's quick start (port, colon, configuration file). The same guide says that without a host specified the HTTP server binds to all interfaces: if the brokers are on an open network, limit the port with the firewall or bind the exporter to an internal interface. The check is a curl on the port, looking for the metric name.

Lag: a number to be read in motion

A group's lag is the difference between the log end offset of each partition and the group's committed offset (the distinction seen in the article on the fundamentals). It is not among the broker metrics you have just mapped. You see it with kafka-consumer-groups.sh --describe --group, you read it on the client side with the records-lag-max metric of kafka.consumer:type=consumer-fetch-manager-metrics, or with an exporter that queries the groups; there is more than one of the latter and I do not recommend one, because I have not verified their maintenance.

Whatever the source, the mistake to avoid is an alarm on an absolute threshold ("more than 50,000 records"). A lag of 100 can be the normal delay of a group that follows closely, and a lag of 100,000 can be a group that is catching up after a stop. What matters is the direction, in relation to the write rate. The model compares three groups over the same interval: written and read are cumulative counters sampled every 30 seconds, with made-up numbers.

import java.util.*;

public class LagTrend {
    // Each sample: records written and records consumed (cumulative) at 30-second intervals.
    static void analizza(String nome, long[] scritti, long[] consumati) {
        long[] lag = new long[scritti.length];
        for (int i = 0; i < lag.length; i++) lag[i] = scritti[i] - consumati[i];       // LEO - offset committed
        long inizio = lag[0], fine = lag[lag.length - 1];
        double prodRate = (scritti[scritti.length - 1] - scritti[0]) / (30.0 * (lag.length - 1));
        double consRate = (consumati[consumati.length - 1] - consumati[0]) / (30.0 * (lag.length - 1));
        String verdetto;
        if (fine > inizio) verdetto = "GROWING: the group cannot keep up";
        else if (fine < inizio) verdetto = "FALLING: catching up, " + (consRate > prodRate
            ? "drained in about " + Math.round(fine / (consRate - prodRate)) + " s" : "");
        else verdetto = "STABLE";
        System.out.printf(Locale.ROOT, "%-26s lag %s -> %s | written %.1f/s, read %.1f/s | %s%n",
            nome, inizio, fine, prodRate, consRate, verdetto);
    }

    public static void main(String[] args) {
        // A and C end with the same lag (100), but one is stable and the other is falling; B has a growing lag.
        analizza("A: same pace", new long[] {1000, 4000, 7000, 10000}, new long[] {900, 3900, 6900, 9900});
        analizza("B: consuming too slowly", new long[] {1000, 4000, 7000, 10000}, new long[] {900, 2900, 4900, 6900});
        analizza("C: catching up", new long[] {1000, 4000, 7000, 10000}, new long[] {200, 3700, 7200, 10000 - 100});
    }
}

// Output:
// A: same pace               lag 100 -> 100 | written 100.0/s, read 100.0/s | STABLE
// B: consuming too slowly    lag 100 -> 3100 | written 100.0/s, read 66.7/s | GROWING: the group cannot keep up
// C: catching up             lag 800 -> 100 | written 100.0/s, read 107.8/s | FALLING: catching up, drained in about 13 s

Group A and group C both end with lag 100, but A is stable and C is falling: reading 107.8 records per second against 100 written, it will drain the remainder in about 13 seconds. Group B has a starting lag identical to A, but reads 66.7 records per second against 100 written and reaches 3100: every 30 seconds it accumulates 1000 more records, and it will not stop on its own. If you want an alarm on lag, base it on the trend (the lag has been growing for N minutes) and, for the absolute level, on a threshold tied to how much time of data you can afford not to have read: the topic's retention decides the point beyond which records are deleted before being read.

Alerts: Prometheus syntax and thresholds

A Prometheus alerting rule has a name, an expression expr, a duration for and, optionally, labels and annotations. The documentation explains that for makes it wait for the expression to remain true for that time before the alert fires: in the meantime it is in the pending state. In the annotations you use {{ $labels.name }} for the values of the labels and {{ $value }} for the value of the expression. The rules below cover the five conditions from the beginning, with only the thresholds that do not depend on load.

# Template to adapt (not run): alerting rules in the format of the Prometheus documentation.
groups:
  - name: kafka.critical
    rules:
      - alert: KafkaBrokerDown
        expr: up{job="kafka-brokers"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Broker {{ $labels.instance }} unreachable"
      - alert: KafkaOfflinePartitions
        expr: sum(kafka_controller_offlinepartitionscount) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $value }} partitions without a leader"
      - alert: KafkaActiveControllers
        expr: sum(kafka_controller_activecontrollercount) != 1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Active controllers: {{ $value }}, it should be 1"
  - name: kafka.warning
    rules:
      - alert: KafkaUnderReplicated
        expr: sum(kafka_server_replicamanager_underreplicatedpartitions) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $value }} partitions with an incomplete ISR for more than 5 minutes"
      - alert: KafkaUnderMinIsr
        expr: sum(kafka_server_replicamanager_underminisrpartitioncount) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Partitions below min.insync.replicas: writes with acks=all fail"

Look at how for is chosen. An active controller other than one and an offline partition are conditions of a few minutes at most, because during a normal election they can last a few seconds: for: 1m or 2m avoids false alarms but does not delay the response. Under-replicated partitions are a warning, not an emergency, until they fall below min.insync.replicas: that is why the rule has for: 5m and warning severity, while the one on UnderMinIsr is critical and short. The times must be tuned to yours: they are a starting point and not a value recommended by the documentation. To check that the rule fires in the case where it is needed, the Prometheus documentation provides unit tests of the rules with promtool, which I could not run here because it is not installed.

Dashboards: the panels, not the identifiers

For ready-to-import Grafana dashboards I write no numeric identifiers: I have not verified them, and a list of unchecked IDs is exactly the kind of data that silently goes stale. Instead, starting from the metrics you already collect, build six panels: the state of each broker (up), under-replicated and under-min-ISR partitions, offline partitions, active controllers, the rate of incoming messages per topic, and the lag per group with the trend. All of them answer a question you should be able to answer in ten seconds: is the cluster serving? is it losing margin? are the consumers keeping up?

What you can do now

You can choose the five metrics that deserve an alarm and the reason for each. You know where Kafka exposes them (JMX MBeans) and how an exporter translates them with pattern rules, in the order in which they are evaluated. You know that lag is read in motion and not in absolute terms, and how the drain time is computed. You can write alerting rules with a sensible for and annotations that state the value. The step after monitoring is sizing the cluster so that the alarms ring rarely, in the article on sizing and disaster recovery; to decide what to do with the messages that cannot be processed, see the article on the dead letter queue. If you come from another point of the series, the links are: Flink and Iceberg.