Skip to main content

Dynamic Labels

Introduction​

When working with Grafana Loki, labels provide a powerful way to categorize and query your logs. While static labels are useful for consistent metadata, dynamic labels take this concept further by allowing you to extract and create labels from log content at query time. This capability enables more flexible and powerful log analysis without changing how your logs are initially ingested.

In this lesson, we'll explore how dynamic labels work in Loki, when to use them, and how they can enhance your log querying workflow.

What Are Dynamic Labels?​

Dynamic labels are labels that are created on-the-fly during query execution, rather than being attached to log entries at ingestion time. They allow you to:

  1. Extract information from log lines into temporary labels
  2. Use these extracted labels for filtering, grouping, and visualization
  3. Perform complex queries that wouldn't be possible with static labels alone

Dynamic labels don't modify your stored log data - they're temporary constructs that exist only for the duration of your query.

Why Use Dynamic Labels?​

Using dynamic labels offers several advantages:

  • Query flexibility: Extract and use information that wasn't labeled at ingestion time
  • Reduced cardinality: Keep your stored label set small while still enabling complex queries
  • Exploratory analysis: Discover patterns in your logs without needing to predict all possible query paths during setup
  • Performance optimization: Balance between query-time extraction and ingestion-time labeling

Creating Dynamic Labels with LogQL​

In Loki's query language (LogQL), dynamic labels are created using the | label_format operator or the newer parse expressions.

Basic Syntax​

The basic syntax for creating a dynamic label is:

{app="frontend"} | label_format new_label=<expression>

Let's explore different ways to create and use dynamic labels:

Method 1: Using label_format​

The label_format operator allows you to create new labels based on existing labels or fixed values.

Example 1: Creating a new label from an existing one​

{app="frontend"} | label_format environment="production"

This adds a new label environment="production" to all log lines matching the query.

Example 2: Modifying an existing label​

{app="frontend", env="prod"} | label_format environment="$env"

This creates a new label environment with the value from the env label.

Example 3: Combining multiple labels​

{app="frontend", region="us-west", datacenter="dc1"} 
| label_format location="$region-$datacenter"

This creates a new location label with the combined values from region and datacenter.

Method 2: Extracting Labels from Log Content​

One of the most powerful uses of dynamic labels is extracting information from the log content itself.

Using Regular Expressions​

{app="payment-service"} 
| regexp `transaction_id=(?P<transaction_id>[a-zA-Z0-9-]+)`

This extracts the transaction ID from logs containing patterns like transaction_id=abc-123 and creates a dynamic label called transaction_id.

Using JSON Parsing​

For JSON-formatted logs:

{app="api"} 
| json 
| label_format user_id="$user.id", request_path="$request.path"

This extracts nested JSON fields into dynamic labels.

Using Pattern Parsing​

For structured logs with consistent patterns:

{app="auth"} 
| pattern `<_> - <method> <endpoint> <status>`
| label_format http_status="$status"

This parses logs matching patterns like 2023-05-10 - GET /users 200 and creates a http_status label.

Using Dynamic Labels​

Once you've created dynamic labels, you can use them in various ways:

Filtering with Dynamic Labels​

{app="api"} 
| json 
| label_format status_code="$response.status"
| status_code=~"5.."

This query:

  1. Selects logs from the "api" application
  2. Parses them as JSON
  3. Creates a dynamic label status_code from the response status
  4. Filters to show only 5xx errors

Grouping and Aggregation​

Dynamic labels are particularly useful for grouping and metrics calculations:

{app="payment-service"}
| regexp `customer_id=(?P<customer_id>[0-9]+)`
| rate(5m)
| sum by (customer_id)

This counts log entries per customer over 5-minute windows.

Real-World Applications​

Let's explore some practical scenarios where dynamic labels shine:

Example 1: Analyzing HTTP Status Codes​

{app="web-server"} 
| pattern `<_> <client_ip> <_> <_> "<method> <path> <_>" <status> <_>`
| label_format http_status="$status", http_path="$path"
| http_status=~"5.."

This extracts HTTP status codes and paths from web server logs, filtering for 5xx errors.

Example 2: Tracking User Behavior​

{app="user-service"} 
| json 
| label_format user_type="$user.type", action="$event.name"
| user_type="premium" and action=~"purchase.*"

This tracks purchase actions by premium users by extracting user type and action from JSON logs.

Example 3: Debugging Microservices​

{app=~"payment.*|order.*"} 
| regexp `trace_id=(?P<trace_id>[a-f0-9]+)`
| trace_id="a1b2c3d4"

This tracks a specific request across multiple microservices by extracting and filtering on a trace ID.

Performance Considerations​

While dynamic labels provide flexibility, they do have performance implications:

  1. Query-time processing: Extracting labels at query time requires additional processing
  2. Regex performance: Complex regular expressions can slow down queries
  3. Data volume: Processing high volumes of logs with dynamic labels can be resource-intensive

Best Practices:​

  • Use dynamic labels for exploratory analysis and specialized queries
  • Consider promoting frequently used dynamic labels to static labels at ingestion time
  • Use specific label matchers before applying regex or JSON parsing to reduce the dataset size
  • Start with simpler extraction methods before using complex regex patterns

Summary​

Dynamic labels provide a powerful way to extract, transform, and utilize information in your logs without modifying your ingestion pipeline. They enable flexible querying, reduce cardinality problems, and support exploratory analysis.

Key points to remember:

  • Dynamic labels are created at query time and don't affect stored data
  • They can be created from existing labels or extracted from log content
  • Methods include label_format, regexp, json, and pattern operations
  • They're useful for filtering, grouping, and visualization
  • Consider performance implications for large-scale production use

Further Exercises​

  1. Extract error codes from your application logs and create visualizations showing error frequency
  2. Use dynamic labels to track user journeys across multiple services
  3. Create a dashboard that uses dynamic labels to show performance metrics by endpoint
  4. Compare query performance with and without dynamic label extraction

Additional Resources​


If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)