Centralized logging and monitoring in .NET Core

Centralized logging and monitoring in .NET Core

To implement centralized logging and monitoring in a .NET Core application, you will typically use a structured logging framework like Serilog, an observability standard like OpenTelemetry, and a centralized platform for log aggregation and visualization, such as Seq or the Elastic Stack.

Architectural overview

The general architecture for centralized logging and monitoring in .NET Core involves three main parts:

• Instrumentation: Within your .NET Core application, you'll use a logging and observability library to capture structured logs, metrics, and traces.
• Collection and Export: The instrumentation library sends this telemetry data to a collector or directly to a centralized server.
• Aggregation and Analysis: A backend system aggregates, stores, and provides a user interface for searching, analyzing, and visualizing the data from all of your application instances.

1. Use a Logging Abstraction (Built-in ILogger)

ASP.NET Core has a built-in ILogger<T> abstraction that works with multiple providers.

This allows you to plug in different backends (console, files, Elasticsearch, Application Insights, etc.) without changing your code.

Example usage:

public class WeatherController : ControllerBase
{
    private readonly ILogger<WeatherController> _logger;

    public WeatherController(ILogger<WeatherController> logger)
    {
        _logger = logger;
    }

    [HttpGet]
    public IActionResult GetWeather()
    {
        _logger.LogInformation("Weather endpoint hit at {time}", DateTime.UtcNow);
        return Ok("Sunny");
    }
}

🔹 2. Structured Logging with Serilog

Instead of plain text logs, use structured logging for better querying and filtering.

Install Serilog

dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Sinks.Console
dotnet add package Serilog.Sinks.Elasticsearch

Configure Program.cs

using Serilog;

var builder = WebApplication.CreateBuilder(args);

// Configure Serilog
Log.Logger = new LoggerConfiguration()
    .Enrich.FromLogContext()
    .WriteTo.Console()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        AutoRegisterTemplate = true,
    })
    .CreateLogger();

builder.Host.UseSerilog();

var app = builder.Build();
app.MapControllers();
app.Run();

Now logs go both to the console and to Elasticsearch, which you can visualize using Kibana.

🔹 3. Centralized Logging Stack

Common stacks for centralization:

• ELK (Elasticsearch, Logstash, Kibana): Store and visualize logs.
• EFK (Elasticsearch, Fluentd/FluentBit, Kibana): Alternative to Logstash.
• Seq: Great for structured logging with Serilog.
• Azure Application Insights: Cloud-based monitoring.
• Grafana + Loki: Popular in Kubernetes setups.

🔹 4. Monitoring & Metrics

Logs alone aren’t enough—you also need metrics and tracing.

Use OpenTelemetry: OpenTelemetry provides distributed tracing, metrics, and logs.

Install packages

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

Configure in Program.cs

using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProvider => tracerProvider
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter(opt =>
        {
            opt.Endpoint = new Uri("http://otel-collector:4317");
        }))
    .WithMetrics(metricsProvider => metricsProvider
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddOtlpExporter(opt =>
        {
            opt.Endpoint = new Uri("http://otel-collector:4317");
        }));

Export data to Prometheus, Jaeger, or Grafana Tempo for visualization.

🔹 5. Error Tracking

Use a dedicated error-tracking service:

• Sentry
• Raygun
• Azure App Insights
• Bugsnag

These provide alerting, dashboards, and user-impact analysis.

🔹 6. Best Practices

• ✅ Structured format: Always log in a structured format (JSON is best).
• ✅ Correlation IDs: Include request IDs and trace IDs for distributed tracing.
• ✅ Data hygiene: Avoid logging sensitive data (PII, passwords, tokens).
• ✅ Environment levels: Configure log levels per environment (e.g., Debug in Dev, Warning/Error in Prod).
• ✅ Retention: Implement log rotation and retention policies.
• ✅ Observability: Use dashboards and alerts in Grafana/Kibana/App Insights.

✅ In short:

• Use ILogger with Serilog for structured logging.
• Send logs to a central store (ELK, Seq, App Insights, Grafana Loki).
• Add OpenTelemetry for tracing & metrics.
• Monitor and alert using Kibana, Grafana, or Azure Monitor.