Logging with Serilog and ELK Stack

Back to Index

Logging in .NET Core with Serilog and ELK Stack

Serilog is a structured logging library for .NET, and the ELK Stack (Elasticsearch, Logstash, Kibana) provides centralized log storage, processing, and visualization.

Setup Example

1. Add Serilog Packages

dotnet add package Serilog
dotnet add package Serilog.Sinks.Console
dotnet add package Serilog.Sinks.Elasticsearch

2. Configure Serilog in Program.cs

Log.Logger = new LoggerConfiguration()
    .Enrich.FromLogContext()
    .WriteTo.Console()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        AutoRegisterTemplate = true,
        IndexFormat = "dotnet-logs-{yyyy-MM}"
    })
    .CreateLogger();

builder.Host.UseSerilog();

3. Docker Compose for ELK Stack

version: '3.9'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
    ports: ['9200:9200']
    environment:
      - discovery.type=single-node

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.1
    ports: ['5601:5601']
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200

📝 Serilog for logging in .NET Core

Serilog is a popular third-party logging library for .NET that focuses on structured logging, meaning it writes log data as a consistent, machine-readable format (e.g., JSON) rather than plain text.

⭐ Key features of Serilog:

Structured logging: Captures rich, searchable data instead of plain strings. Instead of logging a message like "User 123 logged in", you log "User {UserId} logged in", which makes UserId a property that can be easily queried.
Enrichers: Add valuable context to log events, such as the machine name, thread ID, process ID, or environment name.
Sinks: Serilog writes log events to various destinations, including the console, files, databases, and centralized systems like Elasticsearch.
Integration with ILogger: Serilog integrates with the standard .NET ILogger interface, allowing you to use it seamlessly in any ASP.NET Core application.

⚙️ Example: Configuring Serilog with the ELK stack

This example shows how to configure Serilog to send logs directly to Elasticsearch from a .NET Core application, bypassing the need for a separate log collector like Logstash. For most cloud-native scenarios, this is a simpler and more robust approach.

📦 1. Install NuGet packages:

Add the necessary packages to your project.

dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Sinks.Elasticsearch
dotnet add package Serilog.Enrichers.Environment

🧩 2. Configure Serilog in Program.cs:

Set up Serilog to read its configuration from appsettings.json and direct its output to Elasticsearch. This includes adding standard enrichers and handling startup errors gracefully.

🗂️ 3. Define the configuration in appsettings.json:

The appsettings.json file should contain the Serilog configuration, specifying minimum log levels for different namespaces and configuring the Elasticsearch sink.

🧪 4. Use the logger in your application:

Inject ILogger into your classes (like controllers) and use its methods (LogInformation, LogError, etc.) to write structured log messages within your application code.

🧱 ELK stack components

The ELK stack provides the back-end infrastructure for collecting and visualizing your logs. It consists of Elasticsearch, Logstash, and Kibana.

👍 Advantages

Feature	Serilog	ELK Stack
Centralized Logging	Indirectly, by sending logs to a central sink.	Provides a scalable and centralized platform for logs.
Structured Logging	Logs are written in a machine-readable format.	Elasticsearch indexes structured data for powerful search.
Performance	Offers asynchronous logging and batching.	Can handle high log volumes and scale horizontally.
Searchability and Analysis	Enables richer queries.	Kibana provides a user-friendly interface for querying and visualization.

👎 Disadvantages

Feature	Serilog	ELK Stack
Complexity	Configuration and managing sinks can add complexity.	Requires significant setup and maintenance.
Resource Usage	Small performance overhead in high traffic scenarios.	Elasticsearch can be resource-intensive.
Stability	Depends on downstream services; failures can occur if not configured correctly.	Scaling issues and instability can occur with improper configuration.
Cost	Free and open-source.	Managing and scaling infrastructure can incur significant costs.

💡 Best practices and tips

Always use structured logs with message templates, add meaningful enrichers for context, and implement a correlation ID for distributed systems.
Fine-tune log levels for different environments and use a robust sink like Elasticsearch for production instead of console or file logging.
Manage Elasticsearch indices using index lifecycle management (ILM).

⚠️ Precautions

Never log sensitive data and always configure security measures for ELK components.
Avoid logging high cardinality identifiers as they can impact Elasticsearch performance.
Be aware of performance overhead from high log volumes and the dependency on external services like the ELK stack, ensuring robust error handling.

Advantages

Structured logs for better querying
Centralized log storage
Powerful search and filtering
Visual dashboards with Kibana

Disadvantages

Complex setup and configuration
Performance overhead for remote logging
Storage and scaling costs

Best Practices

Use structured logging with context
Set appropriate log levels
Sanitize logs to avoid sensitive data exposure
Use rolling indices in Elasticsearch
Monitor ingestion performance

Precautions

Secure Elasticsearch with TLS and authentication
Restrict access to Kibana dashboards
Use Logstash filters to clean logs

Tips

Tag logs with application name and environment
Combine logs with metrics and traces
Set up alerts in Kibana for anomalies

Back to Index

Prometheus-and-Grafana-Monitoring

Secure API with HTTPS in .NET Core