Using Open Telemetry API in your Microservices

OpenTelemetry is a collection of APIs, SDKs, tools you can use to instrument, generate, capture and export telemetry data. An example of telemetry data are metrics and logs which can then be analyzed to evaluate your application’s performance and behaviour.

In the era of cloud-native applications, it is pretty common to combine polyglot architectures. This leads to a major challenge: how to power observability in a standard way?

The OpenTelemetry project aims to solve this problem by providing a single, vendor-agnostic solution covering the following aspects:

  • A single, open, vendor-agnostic instrumentation library that can be deployed either as as an agent or gateway.
  • An end-to-end implementation to generate, emit, collect, process and export telemetry data.
  • Ability to send data to multiple destinations in parallel through configuration.
  • The ability to support multiple context propagation formats in parallel to assist with migrating as standards evolve.
  • Support for a variety of open-source and commercial protocols, format and context propagation mechanisms as well as providing shims to the OpenTracing and OpenCensus projects.

Before diving into WildFly configuration of OpenTelemetry, we need to clarify two core building blocks used by client applications using Telemetry API_

Span is the building block of a trace and is a named, timed operation that represents a piece of the workflow in the distributed system. Multiple spans are pieced together to create a trace.

A Trace, often viewed as a “tree” of spans, reflects the time that each span started and completed. It also shows you the relationship between spans.

Running Open Telemetry on WildFly

WildFly 25 provides support for OpenTelemetry as you can see from its extension which is available in all server configurations:

<extension module="org.wildfly.extension.opentelemetry"/>

Here is a sample configuration:

<subsystem xmlns="urn:wildfly:opentelemetry:1.0" service-name="example">
    <exporter type="jaeger" endpoint="http://localhost:14250"/>
    <span-processor type="simple" batch-delay="4500" max-queue-size="128" max-export-batch-size="512" export-timeout="4500"/>
    <sampler type="on" ratio="1.0"/>
</subsystem>

The configuration is composed of three core components:

Exporter
The exporter can be selected and configured using the exporter child element, which supports these attributes:

  • exporter: WildFly supports two different exporters:
    • jaeger: The default exporter
    • otlp: The OpenTelemetry protocol
  • endpoint: The URL via which OpenTelemetry will push traces. The default is Jaeger’s endpoint is http://localhost:14250

Span Processor
The span process is configured via the span-processor element, which supports the following attributes:

  • type: The type of span processor to use.
    • batch: The default processor, which sends traces in batches as configured via the remaining attributes
    • simple: Traces are pushed to the exporter as they finish.
  • batch-delay: The amount of time, in milliseconds, to wait before traces are published (default: 5000)
  • max-queue-size: The maximum size of the queue before traces are dropped (default: 2048)
  • max-export-batch-size: The maximum number of traces that are published in each batch, which must be smaller or equal to `max-queue-size (default: 512)
  • export-timeout: The maximum amount of time in milliseconds to allow for an export to complete before being cancelled (default: 30000)

Sampler
The sampler is configured via the sampler element:

  • type: The type of sampler to use
    • on: Always on (all traces are recorded)
    • off: Always off (no traces are recorded)
  • ratio: Return a ratio of the traces (e.g., 1 trace in 10000).

Creating Spans with a basic application

To test OpenTelemetry API, we will be using a simple REST Endpoint which sends some Span data to a target Jaeger server.

@Path("/restendpoint")
public class ExampleEndpoint {
	   @Inject
	    private Tracer tracer;

	    @GET
	    public Response doSomeWork() {
	    	System.out.println("starting.......");
	        final Span span = tracer.spanBuilder("Doing some work")
	                .startSpan();
	        span.makeCurrent();
	        doSomeMoreWork();
	        span.addEvent("Make request to external system.");
	        doSomeMoreWork();
	        span.addEvent("All the work is done.");
	        span.end();

	        return Response.ok().build();
	}

		private void doSomeMoreWork() {			 
		   System.out.println("done");			 			
		}
}

In order to interact with traces, you must first acquire a handle to a Tracer. Next, a trace will automatically be created for each request, and your application code will already be wrapped in a span, which can be used for adding application specific attributes and events.

Also, as you can see, we have wrapped our work items in separate methods. This pattern of wrapping method calls is important, because we always want application code to be able to assume that the current span is correct.

Testing the application

To build the application, we need to include Jakarta EE core libraries and OpenTelemetry dependencies as well:

<dependencyManagement>
	<dependencies>
		<dependency>
			<groupId>io.opentelemetry</groupId>
			<artifactId>opentelemetry-bom</artifactId>
			<version>1.6.0</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	</dependencies>
</dependencyManagement>

<dependencies>
	<dependency>
		<groupId>jakarta.platform</groupId>
		<artifactId>jakarta.jakartaee-api</artifactId>
		<version>8.0.0</version>
		<scope>provided</scope>
	</dependency>
	<dependency>
		<groupId>io.opentelemetry</groupId>
		<artifactId>opentelemetry-api</artifactId>
	</dependency>

</dependencies>

Finally, to capture the OpenTelemetry API we need a Collector server. As said, we will use Jaeger. The simplest way to get Jeager up and running is to launch its Docker image:

docker run -d --name jaeger   -p 6831:6831/udp   -p 5778:5778   -p 14250:14250   -p 14268:14268   -p 16686:16686   jaegertracing/all-in-one:1.16
Please note that our WildFly configuration uses as default the port 14250 therefore we have exported that port in the docker container.

Firstly, test the application from your browser or simply with a cURL:

curl http://localhost:8080/telemetry/rest/restendpoint

Next, head to the Jaeger UI which is available at localhost:16686 and check for the service named “example”. Click on Find Traces.

Finally, you will see the Spans which have been collected so far:

Disabling OpenTracing

By default, both OpenTracing and OpenTelemetry have instrumented the endpoints to enable tracing and export traces to a collector such as Jaeger. In most cases this is not desirable as you will end up with duplicate data on the collector.

To remove OpenTracing from your configuration you can run the following commands from WildFly CLI:

/subsystem=microprofile-opentracing-smallrye:remove()
/extension=org.wildfly.extension.microprofile.opentracing-smallrye:remove()

Conclusion

We had an overview of OpenTelemetry API and how to configure a Jakarta EE application to send spans to an external Collector such as Jaeger.

Special thanks to our Principal software engineer Jason Lee for his help in writing and troubleshooting this tutorial.

The source code for this tutorial is available here: https://github.com/fmarchioni/mastertheboss/tree/master/micro-services/open-telemetry