Prometheus and Grafana Monitoring

Monitoring in .NET Core with Prometheus and Grafana

Prometheus and Grafana are powerful tools for monitoring .NET Core applications. Prometheus collects metrics, and Grafana visualizes them in dashboards.

How It Works

• .NET Core app exposes metrics via /metrics endpoint
• Prometheus scrapes metrics periodically
• Grafana connects to Prometheus and displays dashboards

Setup Example

1. Add Prometheus Metrics Library

dotnet add package prometheus-net.AspNetCore
  

2. Configure Middleware in Program.cs

app.UseRouting();
app.UseEndpoints(endpoints =>
{
    endpoints.MapMetrics(); // exposes /metrics
});
  

3. Prometheus Configuration (prometheus.yml)

scrape_configs:
  - job_name: 'dotnet-app'
    static_configs:
      - targets: ['localhost:5000']
  

4. Grafana Setup

• Connect Grafana to Prometheus as a data source
• Import .NET Core dashboard templates
• Create custom panels for metrics like request rate, latency, and memory usage

Prometheus and Grafana form a powerful, open-source stack for monitoring systems and applications. Prometheus handles the collection, storage, and alerting for time-series metrics, while Grafana serves as the visualization layer, creating customizable dashboards to display and analyze that data. This combination is especially popular for monitoring cloud-native and containerized environments, such as Kubernetes.

Architecture and components

Prometheus:

Prometheus Server: The core component that scrapes (pulls) metrics from configured endpoints and stores them in its time-series database (TSDB).

Time-Series Database (TSDB): The built-in, local storage for metrics, with efficient compression and a configurable data retention policy.

Targets: The applications, servers, or other systems that expose metrics for Prometheus to scrape. Targets can be discovered automatically in dynamic environments like Kubernetes.

Exporters: Lightweight programs that translate metrics from third-party systems (e.g., databases, hardware, APIs) into the format Prometheus understands and exposes them via an HTTP endpoint.

Client Libraries: Language-specific libraries for instrumenting your application code to expose custom, application-specific metrics.

Pushgateway: An intermediary service for collecting metrics from short-lived jobs that cannot be scraped directly by the Prometheus server.

PromQL: A flexible and powerful query language for slicing, dicing, and aggregating the time-series data stored in Prometheus.

Alertmanager: A separate component that handles alerts sent by the Prometheus server, including grouping, silencing, and routing them to notification channels.

Grafana:

Data Source Plugin: A connector that allows Grafana to query data from a source. For this stack, the Prometheus data source plugin connects to the Prometheus server.

Dashboards: Visual canvases made up of panels that display data from one or more data sources. They are highly customizable and can be templated for reusability.

Panels: The individual visualizations on a dashboard, such as graphs, gauges, heatmaps, and tables.

Query Editor: A user interface for writing and executing queries (in this case, PromQL) against the connected data source.

Alerting: Grafana can set its own alert rules based on queries, triggering notifications to services like Slack, email, and PagerDuty.

Transformations: A feature for manipulating data returned by a query before it is visualized, such as combining or filtering data.

Monitoring in .NET Core

You can integrate Prometheus into a .NET Core application using the prometheus-net.AspNetCore NuGet package. This exposes an HTTP endpoint, typically /metrics, for Prometheus to scrape.

Example: Instrumenting an ASP.NET Core API

Add NuGet packages:

//xml
<ItemGroup>
        <PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
</ItemGroup>

Configure the Prometheus Exporter in Program.cs:

//csharp
var builder = WebApplication.CreateBuilder(args);

// ... configure services ...

var app = builder.Build();

// Add Prometheus scraping endpoint
app.UseMetricServer();

// ... configure other middleware ...

app.Run();

Create a custom metric:

//csharp
using Prometheus;

public class HomeController : ControllerBase
{
    private static readonly Counter ApiCallCounter = Metrics
        .CreateCounter("api_calls_total", "Counts API calls");

    [HttpGet]
    public IActionResult Get()
    {
        ApiCallCounter.Inc(); // Increment the counter on each call
        return Ok("Hello, Prometheus!");
    }
}

Configure Prometheus to scrape: In your prometheus.yml file, add a job for your application.

//yml
scrape_configs:
  - job_name: 'my-dotnet-app'
    static_configs:
      - targets: ['localhost:5000'] # Change to your application's host and port

Advantages and disadvantages

Aspect Advantages Disadvantages

Prometheus Multi-dimensional data model: Allows for highly flexible and precise querying using labels.

Pull-based model: More robust in dynamic environments, as Prometheus finds and scrapes targets automatically.

Scalability and Reliability: Each Prometheus server is standalone and autonomous, making it highly reliable during infrastructure failures. Limited long-term storage: Its local TSDB is not optimized for long-term retention of massive datasets without external tooling.

Not for absolute accuracy: The pull model is not suitable for situations requiring 100% data accuracy (e.g., billing).

Grafana Powerful visualizations: Creates highly customizable and interactive dashboards using a variety of panels.

Multi-source integration: Connects with many data sources besides Prometheus, allowing for a unified view of your entire infrastructure.

Rich feature set: Includes advanced features like alerting, templating, and transformations. Requires separate data source: Cannot store data itself and relies on external data sources like Prometheus.

Complexity: Configuring advanced features and custom dashboards can have a steep learning curve.

Best practices and tips

Instrument effectively: Focus on a few key metrics that are meaningful for your application's performance. Avoid collecting excessive or low-value metrics, which can create high cardinality issues in Prometheus.

Use meaningful labels: Name your labels clearly and use them to add context to your metrics (e.g., http_requests_total{method="GET", status="200"}). This makes querying in PromQL much easier.

Leverage templates in Grafana: Use variables to create reusable dashboards that can be easily filtered by different services, environments, or other labels. This prevents dashboard clutter.

Monitor the monitors: Use Prometheus to monitor the health and performance of your Prometheus and Grafana instances.

Use existing resources: Before building custom dashboards, check the vast library of pre-built Grafana dashboards available online. For system-level metrics, use standard exporters like Node Exporter.

Precautions

High cardinality: Be cautious when creating labels with a large number of unique values (e.g., user IDs, session IDs). High-cardinality metrics can overwhelm the Prometheus server and impact performance.

Security: Ensure secure access to your Grafana dashboards with robust user authentication and role-based access controls. For Prometheus, protect your metrics endpoints, as they can expose information about your application.

Data retention: Be mindful of Prometheus's limited data retention. If you need long-term metric storage, consider using a compatible long-term storage solution like Grafana Mimir.

Alerting configuration: Tune your alerts carefully to avoid "alert fatigue" and ensure they trigger for genuinely critical issues.