Using Open Telemetry API in your Microservices

OpenTelemetry is a collection of APIs, SDKs, tools you can use to instrument, generate, capture and export telemetry data. This article shows an example of how you can send and capture telemetry data with a Jakarta EE 10 application server such as WildFly.

In the era of cloud-native applications, it is pretty common to combine polyglot architectures. This leads to a major challenge: how to power observability in a standard way?

The OpenTelemetry project aims to solve this problem by providing a single, vendor-agnostic solution covering the following aspects:

  • A single, open, vendor-agnostic instrumentation library that can be deployed either as as an agent or gateway.
  • An end-to-end implementation to generate, emit, collect, process and export telemetry data.
  • Ability to send data to multiple destinations in parallel through configuration.
  • The ability to support multiple context propagation formats in parallel to assist with migrating as standards evolve.
  • Support for a variety of open-source and commercial protocols, format and context propagation mechanisms as well as providing shims to the OpenTracing and OpenCensus projects.

Before diving into WildFly configuration of OpenTelemetry, we need to clarify two core building blocks used by client applications using Telemetry API_

Span is the building block of a trace and is a named, timed operation that represents a piece of the workflow in the distributed system. You can combine multiple spans to create a trace.

A Trace, often viewed as a “tree” of spans, reflects the time that each span started and completed. It also shows you the relationship between spans.

Running Open Telemetry on WildFly

WildFly 25 provides support for OpenTelemetry as you can see from its extension which is available in all server configurations:

<extension module="org.wildfly.extension.opentelemetry"/>

Here is a sample configuration:

<subsystem xmlns="urn:wildfly:opentelemetry:1.0" service-name="example">
    <exporter type="jaeger" endpoint="http://localhost:14250"/>
    <span-processor type="simple" batch-delay="4500" max-queue-size="128" max-export-batch-size="512" export-timeout="4500"/>
    <sampler type="on" ratio="1.0"/>
</subsystem>

The configuration includes three core components:

Exporter
The Trave exporter, which supports these attributes:

  • exporter: WildFly supports two different exporters:
    • jaeger: The default exporter
    • otlp: The OpenTelemetry protocol
  • endpoint: The URL via which OpenTelemetry will push traces. The default is Jaeger’s endpoint is http://localhost:14250

Span Processor
The Span Processor, which supports the following attributes:

  • type: The type of span processor to use.
    • batch: The default processor, which sends traces in batches as configured via the remaining attributes
    • simple: Traces are pushed to the exporter as they finish.
  • batch-delay: The amount of time, in milliseconds, to wait before publishing traces (default: 5000)
  • max-queue-size: The maximum size of the queue before traces are dropped (default: 2048)
  • max-export-batch-size: The maximum number of traces in each batch, which must be smaller or equal to `max-queue-size (default: 512)
  • export-timeout: The maximum amount of time in milliseconds to allow for an export to complete before timeout (default: 30000)

Sampler
The Sampler, which supports the following element::

  • type: The type of sampler to use
    • on: Always on (Record every trace)
    • off: Always off (Don’t record any trace)
  • ratio: Return a ratio of the traces (e.g., 1 trace in 10000).

Creating Spans with a basic application

To test OpenTelemetry API, we will be using a simple REST Endpoint which sends some Span data to a target Jaeger server.

@Path("/restendpoint")
public class ExampleEndpoint {
	   @Inject
	    private Tracer tracer;

	    @GET
	    public Response doSomeWork() {
	    	System.out.println("starting.......");
	        final Span span = tracer.spanBuilder("Doing some work")
	                .startSpan();
	        span.makeCurrent();
	        doSomeMoreWork();
	        span.addEvent("Make request to external system.");
	        doSomeMoreWork();
	        span.addEvent("All the work is done.");
	        span.end();

	        return Response.ok().build();
	}

		private void doSomeMoreWork() {			 
		   System.out.println("done");			 			
		}
}

In order to interact with traces, you must first acquire a handle to a Tracer. Next, a trace will automatically be created for each request, and your application code will already be wrapped in a Span.

You can annotate Spans with events (Span Events) that can carry zero or more Span Attributes, each of which is itself a key:value map paired automatically with a timestamp.

Testing the application

To build the application, we need to include Jakarta EE core libraries and OpenTelemetry dependencies as well:

<dependencyManagement>
	<dependencies>
		<dependency>
			<groupId>io.opentelemetry</groupId>
			<artifactId>opentelemetry-bom</artifactId>
			<version>1.10.0</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	</dependencies>
</dependencyManagement>

<dependencies>
	<dependency>
		<groupId>jakarta.platform</groupId>
		<artifactId>jakarta.jakartaee-api</artifactId>
		<version>10.0.0</version>
		<scope>provided</scope>
	</dependency>
	<dependency>
		<groupId>io.opentelemetry</groupId>
		<artifactId>opentelemetry-api</artifactId>
	</dependency>

</dependencies>

Finally, to capture the OpenTelemetry API we need a Collector server. As said, we will use Jaeger. The simplest way to get Jeager up and running is to launch its Docker image:

docker run -d --name jaeger   -p 6831:6831/udp   -p 5778:5778   -p 14250:14250   -p 14268:14268   -p 16686:16686   jaegertracing/all-in-one:1.16
Please note that our WildFly configuration uses as default the port 14250 therefore we have exported that port in the docker container.

Firstly, test the application from your browser or simply with a cURL:

curl http://localhost:8080/telemetry/rest/restendpoint

Next, head to the Jaeger UI which is available at localhost:16686 and check for the service named “example”. Click on Find Traces.

Finally, you will see the list of Spans for your service:

Decorating Spans with Attributes

When you send data to the Trace API, you can add custom attributes to spans. For example:

if (activeNetwork.getState() == NetworkState.NO_NETWORK_AVAILABLE) {
tracer.spanBuilder("network.change")
        .setAttribute(NETWORK_STATUS_KEY, "lost")
        .startSpan()
        .setAttribute(
                NET_HOST_CONNECTION_TYPE, activeNetwork.getState().getHumanName())
        .end();
} else {
Span available =
        tracer.spanBuilder("network.change")
                .setAttribute(NETWORK_STATUS_KEY, "available")
                .startSpan();
 
available.end();
}

Consider the following tips when adding attributes:

  • You can add any attribute to a Span. For example: you might add an attribute like user.id so that you could search traces globally for traces containing a specific User.
  • A span can be in multiple categories. For example, external is a more general category than is datastore, so if a span is classified as both external and datastore, it will be indicated as a datastore span in the UI.

Capturing Status and Exceptions

Finally, you can set a status for a Span, typically to specify that a span has not completed successfully – SpanStatus.

span.setStatus(StatusCode.ERROR, "Something wrong happened!");

Besides, if you want to record the specific Exception in a Span, a good idea is to record the Exception in conjunction with setting the Status:

Span span = tracer.spanBuilder("Span#1").startSpan();

try (Scope scope = span.makeCurrent()) {
	// do something
} catch (Throwable throwable) {
  span.setStatus(StatusCode.ERROR, "Something wrong happened!");
  span.recordException(throwable)
} finally {
  span.end(); 
}

Disabling OpenTracing

By default, both OpenTracing and OpenTelemetry have instrumented the endpoints to enable tracing and export traces to a collector such as Jaeger. In most cases this is not desirable as you will end up with duplicate data on the collector.

To remove OpenTracing from your configuration you can run the following commands from WildFly CLI:

/subsystem=microprofile-opentracing-smallrye:remove()
/extension=org.wildfly.extension.microprofile.opentracing-smallrye:remove()

Conclusion

We had an overview of OpenTelemetry API and how to configure a Jakarta EE application to send spans to an external Collector such as Jaeger.

Special thanks to our Principal software engineer Jason Lee for his help in writing and troubleshooting this tutorial.

The source code for this tutorial is available here: https://github.com/fmarchioni/mastertheboss/tree/master/micro-services/open-telemetry