Overview

ROS diagnostics messages are part of jros2client starting from version 10.0.

In this article we cover how users can use ROS diagnostics by subscribing to it with jros2client.

jros2client node will be listening to "/diagnostics_agg" topic for aggregated messages and converting them to OpenTelemetry metrics. All collected metrics will be stored in OpenTelemetry back-end, where they later can be visualized. OpenTelemetry supports multiple back-ends (Prometheus etc). In our example as a back-end we will use Elasticsearch.

As a source of all diagnostics we will use Jetson Orin Nano. All diagnostics from it will be gathered and published to ROS by isaac_ros_jetson_stats

The entire flow looks like this:

Where diagnostics_otel is a Java node we about to implement.

Prereqisites

• Jetson Orin Nano - other Jetson platforms should work as well, as long as they are supported by the isaac_ros_jetson_stats.
• ROS Humble+
• isaac_ros_jetson_stats - following tutorial covers how to install isaac_ros_jetson_stats on Jetson.
• Java 17+
• Elasticsearch is optional, by default all received metrics will be printed to System.out

Java dependencies

List of Java dependencies:

implementation "io.github.lambdaprime:jros2client:10.0"
implementation "io.opentelemetry:opentelemetry-api:1.43.0"

Additionally, to export OpenTelemetry metrics to Elasticsearch:

implementation "io.github.lambdaprime:id.opentelemetry-exporters-pack:4.0"

Code

Configure OpenTelemetry back-end

To export metrics to Elasticsearch we setup PeriodicMetricReader with a duration of every 3 seconds:

/** Setup OpenTelemetry to send metrics to Elasticsearch */
private static void setupMetrics() {
    var metricReader =
        PeriodicMetricReader.builder(
            new ElasticsearchMetricExporter(
                    URI.create("ELASTICSEARCH_URL/diagnostics_otel"),
                    Optional.empty(),
                    Duration.ofSeconds(5),
                    true))
                .setInterval(Duration.ofSeconds(3))
                .build();
    var sdkMeterProvider =
        SdkMeterProvider.builder().registerMetricReader(metricReader).build();
    OpenTelemetrySdk.builder().setMeterProvider(sdkMeterProvider).buildAndRegisterGlobal();
}

Create metrics for Jetson

Here is the list of Jetson stats we will be sending to OpenTelemetry:

List<String> isaac_ros_jetson_stats = List.of(
    "/jtop/CPU/jetson_stats/temp/cpu",
    "/jtop/Board/pwmfan/PWM 0",
    "/jtop/Board/pwmfan/RPM 0",
    "/jtop/GPU/jetson_stats/gpu/gpu",
    "/jtop/GPU/jetson_stats/temp/gpu",
    "/jtop/GPU/gpu/Used",
    "/jtop/Memory/ram/Use");

In order to emit such stats to OpenTelemetry we need to create a Meter:

meter = GlobalOpenTelemetry.getMeter(DiagnosticsOpenTelemetryApp.class.getSimpleName());

As the metric instrument, for emitting Jetson stats, we will use gauges. For each stat we create separate gauge:

Map<String, DoubleGauge> gauges = new HashMap<>();
isaac_ros_jetson_stats.stream()
    .forEach(stat -> gauges.put(stat, meter.gaugeBuilder(toMetricName(stat)).build()));

Subscribe to ROS diagnostics

With all metrics setup in-place, we now ready to subscribe to "/diagnostics_agg":

var configBuilder = new JRos2ClientConfiguration.Builder();
// use configBuilder to override default parameters (network interface, RTPS settings etc)
var client = new JRos2ClientFactory().createClient(
    configBuilder.build());
var topicName = "/diagnostics_agg";

// register a new subscriber with default QOS policies
// users can redefine QOS policies using overloaded version of subscribe() method
client.subscribe(new TopicSubscriber<>(DiagnosticArrayMessage.class, topicName) {
    @Override
    public void onNext(DiagnosticArrayMessage item) {
        System.out.println("New item received");
        try {
            emitToOpenTelemetry(item);
        } finally {
            // request next message
            getSubscription().get().request(1);
        }
    }
});

Emit metrics

Function emitToOpenTelemetry is responsible for converting received ROS diagnostics to OpenTelemetry metrics and emitting them. For each Jetson diagnostic it looks up gauge from "gauges" map and updates it.

void emitToOpenTelemetry(DiagnosticArrayMessage item) {
    for (var s : item.status) {
        var name = s.name.data;
        for (var pair : s.values) {
            var keyName = pair.key.data;
            var fullName = name + "/" + keyName;
            var gauge = gauges.get(fullName);
            if (gauge == null) continue;
            try {
                var val = parseMeasurement(pair.value.data);
                gauge.set(val);
                System.out.println(fullName + "=" + val);
            } catch (Exception e) {
                System.err.format(
                    "Could not parse double value from %s: %s", pair, e.getMessage());
            }
        }
    }
}

Later PeriodicMetricReader emits all gauges values to OpenTelemetry back-end (Elasticsearch).

Run

The complete code can be found in diagnostics_otel project which is part of jros2client examples.

Build diagnostics_otel:

diagnostics_otel can be started right from the Jetson or from any other host which is located in the same network as Jetson.

On Jetson, start isaac_ros_jetson_stats:

Results

To generate load on Jetson we run Mistral 7B using Ollama and asked it couple of questions.

Final metrics in Elasticsearch: