Prometheus Metric Exposition

Introduction

Prometheus collects metrics by scraping HTTP endpoints that expose data in a specific text-based format. This format, known as the Prometheus Exposition Format, is a simple and standardized way to represent time-series data. Understanding this format is essential for both creating custom exporters and instrumenting your applications to work with Prometheus.

In this guide, we'll explore how Prometheus metrics are exposed, the structure of the exposition format, and how to implement metric endpoints in different programming languages.

What is Metric Exposition?

Metric exposition is the process of making metrics available for collection by Prometheus. When Prometheus scrapes a target, it expects to find metrics in a specific text-based format at the configured endpoint (usually /metrics).

The exposition process involves:

1. Collecting relevant data points from your application
2. Formatting those data points according to the Prometheus exposition format
3. Exposing them via an HTTP endpoint
4. Responding to Prometheus scrape requests with the formatted metrics

The Prometheus Exposition Format

The Prometheus exposition format is a text-based format with a simple structure. Let's break it down:

Basic Structure

# HELP <metric_name> <description>
# TYPE <metric_name> <type>
<metric_name>[{<label_name>=<label_value>, ...}] <value> [<timestamp>]

Let's examine each component:

• Comments (HELP and TYPE): Lines starting with # provide metadata about metrics
  • # HELP provides a human-readable description of what the metric represents
  • # TYPE specifies the metric type (counter, gauge, histogram, or summary)
• Metric name: A valid identifier that follows Prometheus naming conventions
• Labels: Optional key-value pairs enclosed in curly braces that add dimensions to your metrics
• Value: The actual numeric value of the metric
• Timestamp: An optional Unix timestamp in milliseconds (rarely used in direct exposition)

Example of Metric Exposition

Here's an example of metrics exposed in the Prometheus format:

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="post",endpoint="/api/users"} 1027
http_requests_total{method="get",endpoint="/api/users"} 8743
# HELP http_request_duration_seconds HTTP request duration in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1"} 21468
http_request_duration_seconds_bucket{le="0.5"} 21870
http_request_duration_seconds_bucket{le="1"} 21962
http_request_duration_seconds_bucket{le="5"} 22000
http_request_duration_seconds_bucket{le="+Inf"} 22000
http_request_duration_seconds_sum 5897.3
http_request_duration_seconds_count 22000

In this example:

• We have two metrics: http_requests_total (a counter) and http_request_duration_seconds (a histogram)
• The counter has different label combinations for different HTTP methods and endpoints
• The histogram includes bucket values, sum, and count

Naming Conventions

Prometheus has specific naming conventions for metrics:

• Metric names should follow the pattern: [a-zA-Z_:][a-zA-Z0-9_:]*
• Names should be descriptive but concise
• Common patterns include:
  • <namespace>_<subsystem>_<name>_<unit> (e.g., http_server_requests_seconds)
  • Use singular for the unit (e.g., seconds not second)
• Label names follow similar restrictions: [a-zA-Z_][a-zA-Z0-9_]*

Implementing a Metrics Endpoint

Now, let's see how to implement a metrics endpoint in different programming languages.

Example in Go

Here's how to expose metrics using the official Prometheus client library for Go:

go

package main

import (
    "net/http"
    "log"
    
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    // Create a counter metric
    requestsCounter := prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint"},
    )
    
    // Register the metric with Prometheus
    prometheus.MustRegister(requestsCounter)
    
    // Increment the counter (this would be in your request handlers)
    requestsCounter.With(prometheus.Labels{"method": "get", "endpoint": "/api/users"}).Inc()
    
    // Expose the metrics via HTTP
    http.Handle("/metrics", promhttp.Handler())
    log.Fatal(http.ListenAndServe(":8080", nil))
}

When accessing the /metrics endpoint, Prometheus would see:

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="get",endpoint="/api/users"} 1

Example in Python

Here's how to expose metrics using the Prometheus client for Python:

python

from prometheus_client import Counter, start_http_server
import time

# Create a counter metric
requests_counter = Counter('http_requests_total', 
                           'Total number of HTTP requests',
                           ['method', 'endpoint'])

# Increment the counter
requests_counter.labels(method='get', endpoint='/api/users').inc()

# Start the HTTP server to expose metrics
start_http_server(8000)

# Keep the application running
while True:
    time.sleep(1)

When accessing the metrics endpoint (default is port 8000), you would see:

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{endpoint="/api/users",method="get"} 1.0

Example in Java

Using the Prometheus client for Java:

java

import io.prometheus.client.Counter;
import io.prometheus.client.exporter.HTTPServer;
import java.io.IOException;

public class MetricsExample {
    public static void main(String[] args) throws IOException {
        // Create a counter metric
        Counter requestsCounter = Counter.build()
            .name("http_requests_total")
            .help("Total number of HTTP requests")
            .labelNames("method", "endpoint")
            .register();
            
        // Increment the counter
        requestsCounter.labels("get", "/api/users").inc();
        
        // Start an HTTP server to expose metrics
        HTTPServer server = new HTTPServer(8000);
        
        // Keep the application running
        while (true) {
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

Content Negotiation

Prometheus uses HTTP content negotiation to support different format versions. The current Prometheus exposition format is version 0.0.4, and Prometheus will include the following header in its scrape requests:

Accept: application/openmetrics-text; version=0.0.1,text/plain;version=0.0.4;q=0.5,*/*;q=0.1

Most client libraries handle this automatically, but if you're implementing a custom exporter, you should return the appropriate content type:

Content-Type: text/plain; version=0.0.4

OpenMetrics Format

OpenMetrics is an evolution of the Prometheus exposition format, now standardized by the CNCF. It's backward compatible with the Prometheus format but adds some enhancements:

• Support for exemplars (sample traces related to metrics)
• Improved timestamp precision
• Support for additional metadata
• More flexible text encoding

Here's an example of OpenMetrics format:

# TYPE http_requests_total counter
# HELP http_requests_total Total number of HTTP requests
http_requests_total{method="get",endpoint="/api/users"} 8743 1623054940000
# EOF

The # EOF marker is unique to OpenMetrics format and signals the end of the metrics output.

Best Practices for Metric Exposition

1. Keep it lightweight: The /metrics endpoint should respond quickly and use minimal resources
2. Use appropriate metric types: Choose counters, gauges, histograms, and summaries appropriately
3. Use labels judiciously: Labels create separate time series, which increases cardinality
4. Follow naming conventions: Use consistent, descriptive names
5. Include helpful metadata: Use HELP comments to make metrics self-documenting
6. Security considerations: Consider adding authentication if metrics contain sensitive data
7. Content compression: For large metric sets, consider enabling HTTP compression

Creating a Custom Exporter

If you need to expose metrics from a system that doesn't natively support Prometheus, you can create a custom exporter. Here's a simple pattern:

1. Collect metrics from the target system (via API, database queries, log parsing, etc.)
2. Transform this data into Prometheus metrics
3. Expose these metrics via an HTTP endpoint

Here's a minimal example in Python for a custom exporter:

python

from prometheus_client import Gauge, start_http_server
import time
import requests

# Create gauge metrics
system_users = Gauge('system_users_total', 'Total number of users in the system')
system_cpu_usage = Gauge('system_cpu_usage_percent', 'CPU usage percentage')

def collect_metrics():
    # In a real exporter, you would fetch this data from your target system
    response = requests.get('http://my-system-api/stats')
    data = response.json()
    
    # Update the Prometheus metrics
    system_users.set(data['user_count'])
    system_cpu_usage.set(data['cpu_percent'])

# Start the HTTP server to expose metrics
start_http_server(8000)

# Periodically collect metrics
while True:
    collect_metrics()
    time.sleep(15)  # Collect every 15 seconds

Implementing Metric Exposition in a Web Application

Let's see a practical example of adding Prometheus metrics to a simple web application:

Example: Flask Web Application with Metrics

python

from flask import Flask, request
from prometheus_client import Counter, Histogram, generate_latest
import time

app = Flask(__name__)

# Create metrics
request_counter = Counter(
    'app_request_count', 
    'Application Request Count',
    ['method', 'endpoint', 'http_status']
)

request_latency = Histogram(
    'app_request_latency_seconds',
    'Application Request Latency',
    ['method', 'endpoint']
)

@app.route('/metrics')
def metrics():
    return generate_latest()

@app.route('/')
def homepage():
    # Record request latency
    start_time = time.time()
    
    # Your application logic here
    result = "Welcome to the homepage!"
    
    # Record metrics
    latency = time.time() - start_time
    request_counter.labels('GET', '/', 200).inc()
    request_latency.labels('GET', '/').observe(latency)
    
    return result

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

In this example:

1. We create two metrics: a counter for request count and a histogram for request latency
2. We implement a /metrics endpoint that generates the Prometheus exposition format
3. For each request, we increment the counter and observe the latency

When Prometheus scrapes the /metrics endpoint, it would see something like:

# HELP app_request_count Application Request Count
# TYPE app_request_count counter
app_request_count{endpoint="/",http_status="200",method="GET"} 12.0

# HELP app_request_latency_seconds Application Request Latency
# TYPE app_request_latency_seconds histogram
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.005"} 8.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.01"} 10.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.025"} 11.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.05"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.075"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.1"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.25"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.5"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="0.75"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="1.0"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="2.5"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="5.0"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="7.5"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="10.0"} 12.0
app_request_latency_seconds_bucket{endpoint="/",method="GET",le="+Inf"} 12.0
app_request_latency_seconds_sum{endpoint="/",method="GET"} 0.05290722846984863
app_request_latency_seconds_count{endpoint="/",method="GET"} 12.0

Data Flow Visualization

Here's a visualization of how metrics flow from your application to Prometheus:

Troubleshooting Metric Exposition

If you're having issues with metric exposition, here are some common problems and solutions:

1. Metrics not showing up in Prometheus

   • Verify your /metrics endpoint is accessible
   • Check Prometheus target status in the Prometheus UI
   • Ensure your metric names follow the correct format

2. High cardinality issues

   • Reduce the number of label combinations
   • Avoid using high-cardinality values like user IDs as labels

3. Slow metric endpoint

   • Optimize metric collection
   • Consider batching metric updates
   • Use a more efficient HTTP server

4. Invalid metric format

   • Validate your metrics output using the promtool
   • Ensure you're following the exposition format specification

Summary

Prometheus metric exposition is the process of making your application metrics available in a format that Prometheus can collect. The Prometheus exposition format is a simple, text-based format that supports different metric types and dimensions through labels.

Key takeaways from this guide:

• The Prometheus exposition format has a specific structure with metadata comments, metric names, labels, and values
• Client libraries are available for most programming languages to help with metric exposition
• You can create custom exporters for systems that don't directly support Prometheus
• Following best practices for naming and labeling metrics is important for maintainability and performance
• The OpenMetrics format is an evolution of the Prometheus format with added capabilities

Additional Resources

• Prometheus Exposition Format Documentation
• OpenMetrics Specification
• Promtool for validating metrics

Exercises

1. Create a simple HTTP server that exposes a counter metric in the Prometheus format
2. Extend a web application with request count and latency metrics
3. Create a custom exporter for a system of your choice (e.g., database, message queue)
4. Use the promtool to validate your metrics exposition format
5. Implement metric exposition with authentication to protect sensitive metrics