Distributed Tracing in Microservices

Distributed Tracing in Microservices

1. What Is Distributed Tracing?

Distributed Tracing is a technique used to track a single request as it travels through multiple services in a distributed application.

In a microservices architecture, a single user request may pass through several services, such as an API Gateway, an Order Service, a Payment Service, and an Inventory Service. Each service performs a specific task, and the request may fail or become slow at any point.

Distributed tracing helps developers understand:

• Which services handled a request.
• How long each service took to process the request.
• Where a request failed.
• Which service is responsible for a performance bottleneck.
• How requests flow between different microservices.

In simple terms: Distributed tracing provides an end-to-end view of a request as it travels across multiple microservices.

2. Why Do We Need Distributed Tracing?

Consider an e-commerce application consisting of the following services:

• API Gateway
• Order Service
• Payment Service
• Inventory Service
• Notification Service

When a customer places an order, the application may execute the following operations:

1. The API Gateway receives the request.
2. The Order Service creates the order.
3. The Payment Service processes the payment.
4. The Inventory Service updates the stock.
5. The Notification Service sends an order confirmation.

Suppose the customer reports that placing an order takes 5 seconds.

Without distributed tracing, developers might need to inspect logs from multiple services and manually correlate them to understand the delay.

With distributed tracing, developers can see the complete request flow and identify which service consumed most of the time.

3. How Distributed Tracing Works

Distributed tracing commonly uses three important concepts: Trace, Span, and Context Propagation.

3.1 Trace

A Trace represents the complete journey of a request across multiple services.

For example, a request to place an order may have a trace ID such as:

TraceId: 4bf92f3577b34da6a3ce929d0e0e4736

The same trace ID identifies related operations throughout the request's journey, allowing developers to view them together.

3.2 Span

A Span represents an individual operation performed as part of a trace.

Examples include:

• Receiving an HTTP request in the API Gateway.
• Creating an order in the Order Service.
• Calling the Payment Service.
• Updating inventory in the Inventory Service.

Each span can contain information such as:

• Operation name.
• Start time and duration.
• Trace ID and span ID.
• Parent span ID, when applicable.
• Attributes describing the operation.
• Status and error information.

3.3 Context Propagation

Context propagation carries trace information from one service to another so that separate operations can be connected to the same trace.

For example, when the Order Service calls the Payment Service, tracing instrumentation can propagate the trace context through HTTP headers.

A commonly used standard is W3C Trace Context. Its HTTP headers include:

• traceparent — carries the trace ID, parent span ID, and tracing flags.
• tracestate — optionally carries vendor-specific tracing information.

When the Payment Service receives the request, its tracing instrumentation uses this context to associate the payment operation with the existing trace.

4. Example of Distributed Tracing Flow

Consider a customer placing an order. The request passes through three services.

The trace might contain the following spans:

Span	Service	Operation	Duration
Span 1	API Gateway	Receive order request	50 ms
Span 2	Order Service	Create order	100 ms
Span 3	Payment Service	Process payment	800 ms
Span 4	Inventory Service	Update stock	150 ms

Note: These are illustrative durations, not measurements from a running application. Spans may overlap when operations execute concurrently, so their durations should not always be added together to calculate the total request time.

From this example, the Payment Service appears to be the largest individual contributor to latency. Developers can investigate payment processing, database queries, or calls to an external payment provider.

5. Distributed Tracing vs. Logging vs. Metrics

Aspect	Logging	Metrics	Distributed Tracing
Purpose	Records individual events and messages.	Measures system performance and health.	Tracks request execution across services.
Example	Payment failed.	Payment API latency is high.	Shows that a request spent most of its time in the Payment Service.
Typical data	Messages, timestamps, exceptions.	Request rate, error rate, latency.	Traces, spans, parent-child relationships, durations.
Best use	Understanding what happened.	Understanding overall system health.	Understanding where a particular request spent its time.

These three observability signals complement one another. For example, a trace can identify a slow payment operation, metrics can show whether payment latency is increasing across the application, and logs can explain the specific error.

6. What Is OpenTelemetry?

OpenTelemetry (OTel) is an open-source observability framework used to generate, collect, and export telemetry data, including traces, metrics, and logs.

In .NET applications, OpenTelemetry provides instrumentation libraries that help developers collect distributed tracing information without manually creating every span.

Common components include:

• Instrumentation: Captures operations such as incoming HTTP requests and outgoing HTTP calls.
• Activity: The .NET mechanism commonly used to represent trace spans.
• ActivitySource: Allows application code to create custom activities.
• OpenTelemetry SDK: Configures collection, processing, sampling, and exporting of telemetry.
• Exporter: Sends telemetry to a supported destination.
• Collector: An optional component that receives, processes, and forwards telemetry.

Common tracing backends include Jaeger, Grafana Tempo, and Zipkin. Telemetry can also be sent to commercial observability platforms.

7. Implement Distributed Tracing in C#

Let's build a simple example using ASP.NET Core Web API, HttpClient, and OpenTelemetry.

We will create two microservices:

• Order Service: Receives an order request and calls the Payment Service.
• Payment Service: Processes the payment request and returns a response.

When a request reaches the Order Service, tracing instrumentation creates a server span. When the Order Service calls the Payment Service using an instrumented HttpClient, the tracing system creates a client span and propagates the trace context to the Payment Service.

The Payment Service creates its own server span, which is connected to the same trace.

Step 1: Create the Projects

Create two ASP.NET Core Web API projects:

dotnet new webapi -n OrderService
dotnet new webapi -n PaymentService

Use a compatible, supported .NET SDK and target framework for both projects.

Step 2: Install OpenTelemetry Packages

In both projects, install the following packages:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.Console

The ASP.NET Core instrumentation captures incoming HTTP requests, while the HTTP instrumentation captures outgoing requests made through HttpClient. The console exporter is useful for demonstrating tracing locally.

Step 3: Configure the Order Service

Replace the contents of OrderService/Program.cs with the following example:

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHttpClient("PaymentClient", client =>
{
    client.BaseAddress = new Uri("http://localhost:5002");
});

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource =>
        resource.AddService("OrderService"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter());

var app = builder.Build();

app.MapGet("/orders/{orderId:int}",
    async (int orderId, IHttpClientFactory clientFactory) =>
{
    var client = clientFactory.CreateClient("PaymentClient");

    var payment = await client.GetFromJsonAsync<PaymentResponse>(
        $"/payments/{orderId}");

    return Results.Ok(new
    {
        OrderId = orderId,
        Status = "Order processed",
        Payment = payment
    });
});

app.Run();

record PaymentResponse(int OrderId, string Status);

What happens here?

• AddAspNetCoreInstrumentation() instruments incoming requests to the Order Service.
• AddHttpClientInstrumentation() instruments outgoing HTTP calls made through HttpClient.
• AddConsoleExporter() writes collected spans to the console.
• AddService("OrderService") identifies the service in telemetry.
• GetFromJsonAsync() calls the Payment Service and deserializes its JSON response.

The PaymentResponse record represents the response expected from the Payment Service.

Step 4: Configure the Payment Service

Replace the contents of PaymentService/Program.cs with the following example:

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource =>
        resource.AddService("PaymentService"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddConsoleExporter());

var app = builder.Build();

app.MapGet("/payments/{orderId:int}", (int orderId) =>
{
    return Results.Ok(new
    {
        OrderId = orderId,
        Status = "Payment successful"
    });
});

app.Run();

What happens here?

• The Payment Service uses ASP.NET Core instrumentation to trace incoming HTTP requests.
• The service identifies itself as PaymentService.
• The console exporter displays the collected spans.
• When the Order Service sends a correctly instrumented request with trace context, the Payment Service can connect its server span to the existing trace.

Step 5: Run Both Services

Configure the Order Service to listen on port 5001 and the Payment Service to listen on port 5002. For a local demonstration, you can use the following commands from the respective project directories:

Start the Payment Service:

dotnet run --urls http://localhost:5002

Start the Order Service in a second terminal:

dotnet run --urls http://localhost:5001

Make sure the configured URLs do not conflict with other running applications.

Step 6: Send a Request

Open the following endpoint in a browser or call it using a tool such as curl:

curl http://localhost:5001/orders/101

The Order Service calls the Payment Service, and both services emit tracing information to their respective consoles.

The response will look similar to this:

{
  "orderId": 101,
  "status": "Order processed",
  "payment": {
    "orderId": 101,
    "status": "Payment successful"
  }
}

The exact JSON formatting and property casing may vary depending on the application's serialization configuration.

8. How Are the Two Services Connected in the Trace?

The tracing instrumentation automatically propagates the W3C trace context through the outgoing HTTP request when supported by the configured instrumentation.

The logical trace looks like this:

Trace: Customer places order
|
|-- Span 1: OrderService
|      Operation: GET /orders/101
|
|      |-- Span 2: HTTP client call
|             Operation: GET /payments/101
|
|             |-- Span 3: PaymentService
|                    Operation: GET /payments/101

The spans are related through trace context and parent-child relationships. The outgoing HTTP client span and the receiving server span represent different sides of the same HTTP call.

Important: Each service does not need to generate a separate unrelated trace. When context propagation is configured correctly, both services participate in the same distributed trace.

9. View Traces in Jaeger

The console exporter is useful for learning, but it is not ideal for investigating complex traces across many services.

For a visual trace view, you can export telemetry to a tracing backend such as Jaeger or Grafana Tempo. A common production setup uses the OpenTelemetry Protocol (OTLP) to send telemetry to an OpenTelemetry Collector, which then forwards it to a tracing backend.

A typical architecture is:

To use Jaeger with this example, replace or supplement the console exporter with an appropriate OTLP exporter and configure the application's OTLP endpoint to point to the collector or tracing backend. Configure the backend and collector to receive the selected protocol and port.

In the tracing interface, you can search for a trace and inspect:

• The services involved in the request.
• The parent-child span hierarchy.
• The duration of each operation.
• Errors and status information.
• Attributes associated with each span.

10. Creating Custom Spans in C#

Automatic instrumentation captures common HTTP operations. However, you may also want to trace important business operations such as validating an order or calculating a shipping fee.

You can create custom spans using ActivitySource and Activity.

For example:

using System.Diagnostics;

public class OrderProcessor
{
    private static readonly ActivitySource ActivitySource =
        new("OrderService.OrderProcessor");

    public void ProcessOrder(int orderId)
    {
        using var activity =
            ActivitySource.StartActivity("ProcessOrder");

        activity?.SetTag("order.id", orderId);

        // Perform order processing logic.
    }
}

In this example, a custom span named ProcessOrder is created when the ProcessOrder() method is called. The span includes an attribute order.id to help identify the specific order being processed.

To collect this custom span, register its activity source with OpenTelemetry:

.WithTracing(tracing=> tracing
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddSource("OrderService.OrderProcessor")
    .AddConsoleExporter())
 

Include AddSource("OrderService.OrderProcessor") in the existing tracing configuration rather than creating a second independent configuration.

The custom span is created when ProcessOrder() executes. If it runs within an existing traced request, it can become part of that trace.

The order.id attribute helps identify the business operation, but production systems should avoid putting sensitive information or high-cardinality identifiers into metric labels. Consider your observability backend's data-retention and access policies when recording business identifiers.

11. How to Handle Errors in Distributed Tracing

Distributed tracing helps identify failures across service boundaries.

For example, if the Payment Service returns an HTTP 500 response, the trace can help identify:

• The request that failed.
• The service that returned the error.
• The duration before the failure occurred.
• The upstream service that called the failing service.
• Relevant error information recorded by instrumentation or application code.

For HTTP instrumentation, an unsuccessful HTTP response is not necessarily represented as an exception. The span status and recorded error information depend on the instrumentation version and configuration.

For business operations that fail, you can explicitly record an exception and set an error status on a custom activity where appropriate.

12. Distributed Tracing in Asynchronous Microservices

Microservices frequently communicate through message brokers such as RabbitMQ, Apache Kafka, or Azure Service Bus.

For example:

In this scenario, the producer should inject trace context into the message headers or properties, and the consumer should extract that context when processing the message.

Appropriate messaging instrumentation or manual propagation can connect the producer and consumer operations. Depending on the messaging pattern, the consumer span may be a child of the producer span or may be linked to it.

This allows developers to investigate the processing of an order even when the services do not communicate directly through HTTP.

13. Best Practices for Distributed Tracing

• Use OpenTelemetry: Prefer standard instrumentation and protocols instead of building a custom tracing framework.
• Use consistent service names: Assign meaningful service names such as OrderService and PaymentService.
• Propagate trace context: Ensure HTTP and messaging calls carry tracing context across service boundaries.
• Use sampling: Control the amount of telemetry collected in high-traffic environments.
• Record meaningful attributes: Include useful, non-sensitive information that helps explain an operation.
• Avoid sensitive data: Do not record passwords, access tokens, payment card details, or unnecessary personal information.
• Correlate logs and traces: Include trace and span identifiers in structured logs when supported by the logging configuration.
• Monitor errors and latency: Use tracing together with metrics and logs.
• Use a suitable backend: Choose a tracing platform that supports your traffic volume, retention requirements, and operational needs.
• Consider data overhead: Exporting every span from every request can increase storage and processing costs.

14. Common Mistakes

• Missing context propagation: Services may produce separate traces if the trace context is not propagated.
• Instrumenting only one service: The trace may not show the complete request journey.
• Using only console output in production: Console output is useful for learning but is difficult to search and analyze at scale.
• Recording too much data: Excessive span attributes can increase storage costs and expose sensitive information.
• Assuming tracing fixes performance problems: Tracing identifies where time is spent; developers must still investigate and fix the underlying cause.
• Ignoring asynchronous messaging: Trace context must also be propagated through message brokers when end-to-end tracing is required.

15. Interview Questions and Answers

Q1. What is distributed tracing?

Distributed tracing is an observability technique that tracks a request across multiple services using a trace ID and related spans.

Q2. What is the difference between a trace and a span?

A trace represents the complete request journey, while a span represents an individual operation within that journey.

Q3. What is context propagation?

Context propagation transfers trace information between services so that operations performed by different services can be connected to the same trace.

Q4. What is OpenTelemetry?

OpenTelemetry is an open-source framework for generating, collecting, and exporting observability data, including distributed traces.

Q5. How do you implement distributed tracing in ASP.NET Core?

Use OpenTelemetry with ASP.NET Core and HttpClient instrumentation, configure an exporter, and ensure trace context is propagated across service calls.

Q6. How does distributed tracing help troubleshoot performance issues?

It shows the duration of operations across services, helping developers identify slow dependencies, expensive operations, and bottlenecks.

Q7. How do you trace requests through RabbitMQ or Kafka?

Propagate trace context through message headers or properties and use suitable producer and consumer instrumentation to connect message-processing spans.

Q8. Does distributed tracing replace logging?

No. Tracing shows the flow and timing of operations, while logging provides detailed event messages and diagnostic information. They work best together.

16. Summary

Distributed tracing is essential for observing requests in microservices architectures where a single operation may involve many independent services.

• Trace: Represents the complete request journey.
• Span: Represents an individual operation.
• Context propagation: Connects operations across service boundaries.
• OpenTelemetry: Provides standard instrumentation and telemetry collection.
• Tracing backend: Stores and visualizes trace data for troubleshooting.

In .NET, ASP.NET Core instrumentation, HttpClient instrumentation, and custom ActivitySource spans provide a practical foundation for implementing distributed tracing. For production systems, combine tracing with structured logging, metrics, appropriate sampling, and a suitable telemetry backend.

Comments and Discussion