Implementing OpenTelemetry in .NET 9: A Complete Guide

OpenTelemetry has reached stable status across all signals (Logs, Metrics, and Traces) in .NET, making it the gold standard for observability in modern applications. This comprehensive guide will walk you through implementing OpenTelemetry in .NET 9 applications with practical examples and best practices.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that provides a unified way to collect, process, and export telemetry data (metrics, logs, and traces) from your applications. It's vendor-neutral and supports multiple programming languages and platforms.

Getting Started with OpenTelemetry in .NET 9

Installing Required Packages

First, let's install the necessary NuGet packages for a typical ASP.NET Core application:

<PackageReference Include="OpenTelemetry" Version="1.9.0" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.9.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.9.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.0.0-beta.12" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />

Basic Configuration

Here's how to configure OpenTelemetry in your Program.cs file using the modern minimal hosting model:

using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// Configure OpenTelemetry
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("MyWebApi", "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment"] = builder.Environment.EnvironmentName,
            ["service.instance.id"] = Environment.MachineName
        }))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddOtlpExporter())
    .WithLogging(logging => logging
        .AddOtlpExporter());

var app = builder.Build();

app.Run();

Advanced Configuration Patterns

Environment-Specific Configuration

In production applications, you'll want different configurations for different environments:

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: builder.Configuration["OpenTelemetry:ServiceName"] ?? "MyWebApi",
            serviceVersion: builder.Configuration["OpenTelemetry:ServiceVersion"] ?? "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment"] = builder.Environment.EnvironmentName,
            ["service.instance.id"] = Environment.MachineName,
            ["k8s.cluster.name"] = builder.Configuration["OpenTelemetry:ClusterName"] ?? "unknown"
        }))
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation(options =>
            {
                options.RecordException = true;
                options.Filter = httpContext => 
                    !httpContext.Request.Path.Value?.Contains("/health") ?? true;
            })
            .AddHttpClientInstrumentation(options =>
            {
                options.RecordException = true;
                options.FilterHttpRequestMessage = request =>
                    !request.RequestUri?.AbsolutePath.Contains("/health") ?? true;
            })
            .AddEntityFrameworkCoreInstrumentation(options =>
            {
                options.SetDbStatementForText = true;
                options.SetDbStatementForStoredProcedure = true;
            });

        // Configure different exporters based on environment
        if (builder.Environment.IsDevelopment())
        {
            tracing.AddConsoleExporter();
        }
        else
        {
            tracing.AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri(builder.Configuration["OpenTelemetry:OtlpEndpoint"] ?? 
                                          "http://localhost:4317");
            });
        }
    });

Custom Instrumentation

Sometimes you need to create custom spans and metrics for your business logic:

using System.Diagnostics;
using System.Diagnostics.Metrics;

public class OrderService
{
    private static readonly ActivitySource ActivitySource = new("MyWebApi.OrderService");
    private static readonly Meter Meter = new("MyWebApi.OrderService");
    private static readonly Counter<int> OrdersProcessedCounter = 
        Meter.CreateCounter<int>("orders_processed_total", "Number of orders processed");
    private static readonly Histogram<double> OrderProcessingDuration = 
        Meter.CreateHistogram<double>("order_processing_duration", "ms", "Order processing duration");

    public async Task<Order> ProcessOrderAsync(int orderId)
    {
        using var activity = ActivitySource.StartActivity("process_order");
        activity?.SetTag("order.id", orderId);
        
        var stopwatch = Stopwatch.StartNew();
        
        try
        {
            // Simulate order processing
            await Task.Delay(Random.Shared.Next(100, 500));
            
            var order = new Order { Id = orderId, Status = "Processed" };
            
            activity?.SetTag("order.status", order.Status);
            activity?.SetStatus(ActivityStatusCode.Ok);
            
            OrdersProcessedCounter.Add(1, new("status", "success"));
            
            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            
            OrdersProcessedCounter.Add(1, new("status", "error"));
            
            throw;
        }
        finally
        {
            stopwatch.Stop();
            OrderProcessingDuration.Record(stopwatch.ElapsedMilliseconds);
        }
    }
}

Registering Custom Sources

Don't forget to register your custom ActivitySource and Meter in the OpenTelemetry configuration:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("MyWebApi.OrderService") // Register custom ActivitySource
        .AddAspNetCoreInstrumentation()
        // ... other instrumentations
    )
    .WithMetrics(metrics => metrics
        .AddMeter("MyWebApi.OrderService") // Register custom Meter
        .AddAspNetCoreInstrumentation()
        // ... other instrumentations
    );

Database Integration

Entity Framework Core Integration

For Entity Framework Core applications, the instrumentation provides rich database telemetry:

builder.Services.AddDbContext<ApplicationDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("DefaultConnection")));

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddEntityFrameworkCoreInstrumentation(options =>
        {
            // Include the actual SQL in spans (use with caution in production)
            options.SetDbStatementForText = true;
            options.SetDbStatementForStoredProcedure = true;
            
            // Enrich spans with additional context
            options.EnrichWithIDbCommand = (activity, command) =>
            {
                activity.SetTag("db.command_timeout", command.CommandTimeout);
            };
        }));

Configuration Best Practices

Resource Configuration

Properly configure your service resources to make telemetry data more useful:

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "MyWebApi",
            serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString() ?? "1.0.0",
            serviceInstanceId: Environment.MachineName)
        .AddAttributes(new Dictionary<string, object>
        {
            // Environment information
            ["deployment.environment"] = builder.Environment.EnvironmentName,
            ["host.name"] = Environment.MachineName,
            ["process.pid"] = Environment.ProcessId,
            
            // Application information
            ["service.namespace"] = "MyCompany.Services",
            ["service.team"] = "Platform Team",
            
            // Deployment information
            ["deployment.version"] = builder.Configuration["DEPLOYMENT_VERSION"] ?? "unknown",
            ["git.commit.sha"] = builder.Configuration["GIT_COMMIT_SHA"] ?? "unknown"
        }));

Sampling Configuration

Configure sampling to control the volume of trace data:

using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new TraceIdRatioBasedSampler(0.1)) // Sample 10% of traces
        // For more complex sampling:
        // .SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.1)))
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter());

Production Considerations

Performance Impact

OpenTelemetry is designed to be lightweight, but consider these performance tips:

Use appropriate sampling rates - Don't trace 100% in high-traffic production
Filter out health check endpoints - They add noise and consume resources
Be cautious with SQL statement recording - Can contain sensitive data and increase overhead
Use batch exporters - More efficient than individual exports

Security Considerations

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            // Don't record request/response bodies by default
            options.RecordException = true;
            
            // Filter sensitive endpoints
            options.Filter = httpContext =>
            {
                var path = httpContext.Request.Path.Value?.ToLowerInvariant();
                return !path?.Contains("/admin") == true && 
                       !path?.Contains("/internal") == true;
            };
            
            // Enrich with safe headers only
            options.EnrichWithHttpRequest = (activity, request) =>
            {
                activity.SetTag("http.request.content_type", request.ContentType);
                // Don't add Authorization header or other sensitive headers
            };
        }));

Integration with .NET Aspire

If you're using .NET Aspire, OpenTelemetry integration is even simpler:

// In your AppHost project
var builder = DistributedApplication.CreateBuilder(args);

var apiService = builder.AddProject<Projects.MyWebApi>("apiservice");

builder.Build().Run();

// In your service project, OpenTelemetry is automatically configured
// You just need to add your custom instrumentation:
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("MyWebApi.OrderService"))
    .WithMetrics(metrics => metrics
        .AddMeter("MyWebApi.OrderService"));

Troubleshooting Common Issues

Missing Traces

If you're not seeing traces:

Verify your ActivitySource name matches the registration
Check sampling configuration
Ensure OTLP endpoint is correct
Look for export errors in logs

High Memory Usage

If experiencing memory issues:

Reduce sampling rate
Configure batch export settings
Filter out high-frequency operations
Review custom attribute cardinality

Conclusion

OpenTelemetry in .NET 9 provides a robust, standards-based approach to observability. By following these patterns and best practices, you can gain valuable insights into your application's behavior while maintaining good performance characteristics.

The key is to start simple with automatic instrumentation and gradually add custom telemetry where it provides business value. Remember to consider the performance and security implications of your observability strategy, especially in production environments.