Prometheus Labels

Introduction

Labels are a fundamental concept in Prometheus that transform simple metrics into powerful, multi-dimensional data points. Think of labels as key-value pairs attached to metrics that provide additional context and enable sophisticated filtering, grouping, and analysis.

Unlike traditional monitoring systems that might create separate metrics for each server or service, Prometheus uses a single metric with different label combinations. This approach dramatically increases the flexibility and power of your monitoring system while keeping the data model clean and intuitive.

Understanding Labels in Prometheus

What Are Labels?

Labels in Prometheus are key-value pairs associated with time series data. They allow a single metric to represent multiple related measurements by varying the label values.

For example, instead of creating separate metrics like http_requests_server_a, http_requests_server_b, and so on, Prometheus uses a single metric http_requests_total with a label server that can have values like a, b, etc.

Label Syntax

Labels follow a simple syntax:

metric_name{label_name="label_value", another_label="another_value"}

For example:

http_requests_total{method="GET", endpoint="/api/users", status="200"}

This represents the total number of HTTP GET requests to the /api/users endpoint that returned a 200 status code.

Types of Labels

Prometheus uses several types of labels:

1. Instrumentation Labels (Manual Labels)

These are labels that you define in your application code when instrumenting your metrics. They represent characteristics that are known at instrumentation time.

// Go example using the Prometheus client library
httpRequestsCounter := prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total number of HTTP requests",
    },
    []string{"method", "endpoint", "status"}, // Label names
)

// Incrementing the counter with specific label values
httpRequestsCounter.WithLabelValues("GET", "/api/users", "200").Inc()

2. Target Labels

These labels are automatically attached to metrics based on their scrape configuration:

scrape_configs:
  - job_name: 'api-server'
    static_configs:
      - targets: ['api.example.com:9090']
        labels:
          environment: 'production'
          region: 'us-west'

This configuration would add job="api-server", environment="production", and region="us-west" labels to all metrics scraped from that target.

3. Generated Labels

These are automatically created by Prometheus during the scrape process:

• job: The configured job name that scraped the metric
• instance: The <host>:<port> of the target being scraped

Best Practices for Using Labels

1. Keep Cardinality Under Control

Cardinality refers to the number of possible label value combinations. High cardinality can severely impact Prometheus performance.

Bad practice:

http_requests_total{user_id="12345", ...}

Using user_id as a label could create millions of time series, overwhelming your system.

Good practice:

http_requests_total{service="user-api", endpoint="/users", method="GET", ...}

2. Use Consistent Label Names

Maintain consistency in label naming across your applications and services:

# Good - consistent label naming
http_requests_total{service="auth", environment="production", ...}
http_errors_total{service="auth", environment="production", ...}

3. Design Labels for Querying

Choose labels that will be useful for querying and filtering metrics:

# Labels enable queries like:
sum(rate(http_requests_total{environment="production", status=~"5.."}[5m])) by (service)

This query calculates the rate of 5xx errors in production, grouped by service.

Working with Labels in PromQL

Labels make the Prometheus Query Language (PromQL) incredibly powerful. Here are some common patterns:

Filtering by Label Values

# All HTTP requests to the /api/users endpoint
http_requests_total{endpoint="/api/users"}

# All HTTP requests with 4xx status codes
http_requests_total{status=~"4.."}

Aggregating by Labels

# Count of HTTP requests by endpoint
sum(http_requests_total) by (endpoint)

# 95th percentile request duration by service and method
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (service, method, le))

Combining Different Metrics with the Same Labels

# Calculate error ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) 
  / 
sum(rate(http_requests_total[5m])) by (service)

Practical Example: Multi-Environment Monitoring

Let's implement a comprehensive example for a web application deployed across multiple environments.

Step 1: Instrument the Application

# Python example
from prometheus_client import Counter, Histogram

# Define metrics with appropriate labels
http_requests = Counter(
    'http_requests_total', 
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

http_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration in seconds',
    ['method', 'endpoint'],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
)

# Usage in a request handler
def handle_request(method, endpoint):
    start = time.time()
    # Process request...
    status = "200"  # Assuming success
    duration = time.time() - start
    
    # Update metrics with labels
    http_requests.labels(method=method, endpoint=endpoint, status=status).inc()
    http_duration.labels(method=method, endpoint=endpoint).observe(duration)

Step 2: Configure Prometheus Scraping

scrape_configs:
  - job_name: 'web-app'
    static_configs:
      - targets: ['app-server-prod-1:9090', 'app-server-prod-2:9090']
        labels:
          environment: 'production'
          region: 'us-east'
      - targets: ['app-server-staging:9090']
        labels:
          environment: 'staging'
          region: 'us-east'

Step 3: Create PromQL Queries Using Labels

# Request rate by environment
sum(rate(http_requests_total[5m])) by (environment)

# Error rate by endpoint in production
sum(rate(http_requests_total{environment="production", status=~"5.."}[5m])) by (endpoint)

# 95th percentile request duration by environment and endpoint
histogram_quantile(0.95, 
  sum(
    rate(http_request_duration_seconds_bucket[5m])
  ) by (environment, endpoint, le)
)

Visualizing Labels in Grafana

Labels make it easy to create dynamic dashboards in Grafana. For example:

1. Create a dashboard variable for environment:

   • Name: environment
   • Query: label_values(http_requests_total, environment)

2. Use the variable in your queries:

   sum(rate(http_requests_total{environment="$environment"}[5m])) by (endpoint)

This allows users to switch between environments in a dropdown, dynamically updating all graphs.

Relabeling in Prometheus

Sometimes you need to modify, add, or filter labels before they're stored. Prometheus offers two mechanisms:

1. relabel_configs

Applied before scraping metrics:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

2. metric_relabel_configs

Applied after scraping but before ingestion:

scrape_configs:
  - job_name: 'web-app'
    static_configs:
      - targets: ['app-server:9090']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'temp_.*'
        action: drop

Common Label Patterns in Real-World Deployments

For HTTP/API Monitoring:

• method (GET, POST, etc.)
• endpoint or path (/api/users, etc.)
• status (200, 404, 500, etc.)
• service or component

For Infrastructure Monitoring:

• instance (hostname:port)
• job (service name)
• environment (production, staging, etc.)
• region or datacenter

For Kubernetes Monitoring:

• namespace
• pod
• container
• node
• deployment

Summary

Labels are what make Prometheus a truly powerful monitoring system. They transform simple metrics into multi-dimensional data points that can be filtered, grouped, and analyzed in countless ways. By attaching contextual information as labels, you create a flexible data model that scales with your infrastructure and provides deep insights into your systems.

Key takeaways:

1. Labels add dimensions to your metrics without creating new metric names
2. They enable powerful filtering and aggregation in PromQL
3. They help control metric cardinality when used properly
4. Well-designed labels make metrics more useful for troubleshooting and analysis
5. Labels support dynamic dashboards in visualization tools like Grafana

Further Learning

To strengthen your understanding of Prometheus labels, try these exercises:

1. Instrument a sample application with at least three different metrics, each with appropriate labels
2. Write PromQL queries that filter and aggregate your metrics using different label combinations
3. Create a Grafana dashboard with variables based on your metric labels
4. Experiment with relabeling to transform or filter your metrics

By mastering labels, you'll unlock the full potential of Prometheus as a monitoring system and gain deeper insights into your applications and infrastructure.