Flow Example

Example Configuration

In this configuration:

• API Request Metrics: The flow captures the size of each API call (api_call_size) and categorizes the data into histogram buckets. Metrics are labeled with http_method, url, and consumer_tag to allow detailed tracking.
• API Response Metrics: The flow collects the remaining rate limits from API responses using the X-Ratelimit-Remaining header, stored as a gauge metric with the url as a label.

Flow Configuration

/etc/lunar-proxy/flows/flow.yaml

name: CustomMetricsCollector

filter:
  url: "*" # Capture metrics for all URLs

processors:
  RequestMetrics: # Collect custom metrics for API requests
    processor: UserDefinedMetrics
    parameters:
      - key: metric_name
        value: "call_size" # Name of the metric to track
      - key: metric_type
        value: "histogram" # Metric type (e.g., histogram)
      - key: metric_value
        value: "api_call_size" # Metric value based on request size
      - key: labels
        value:
          - "http_method" # Label for HTTP method
          - "url" # Label for URL
          - "consumer_tag" # Label for consumer tag
      - key: buckets
        value: # Define the histogram buckets
          - 0
          - 100
          - 250
          - 500
          - 1000
          - 2500
          - 5000
          - 10000

  ResponseMetrics: # Collect custom metrics for API responses
    processor: UserDefinedMetrics
    parameters:
      - key: metric_name
        value: "rate_limit_remaining" # Metric to track remaining rate limits
      - key: metric_type
        value: "gauge" # Metric type (e.g., gauge)
      - key: metric_value
        value: "$[\"headers\"][\"X-Ratelimit-Remaining\"]" # JSON path for rate limit from headers
      - key: labels
        value:
          - "url" # Label for URL
      - key: buckets
        value: [] # No buckets needed for gauge

flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start # Start of the request flow
      to:
        processor:
          name: RequestMetrics # Apply custom request metrics

    - from:
        processor:
          name: RequestMetrics # After collecting metrics
      to:
        stream:
          name: globalStream
          at: end # End of the request flow

  response:
    - from:
        stream:
          name: globalStream
          at: start # Start of the response flow
      to:
        processor:
          name: ResponseMetrics # Apply custom response metrics

    - from:
        processor:
          name: ResponseMetrics # After collecting response metrics
      to:
        stream:
          name: globalStream
          at: end # End of the response flow

Flow Parameters

Parameter	Description	Type	Example
name	The name of the flow.	String	CustomMetricsCollector
filter.url	URL pattern for filtering traffic.	String	* (All URLs)
processors.RequestMetrics.metric_name	Name of the custom metric for requests.	String	call_size
processors.RequestMetrics.metric_type	Type of the metric (e.g., counter, gauge, histogram).	String	histogram
processors.RequestMetrics.metric_value	Value for the metric, typically a request property like size.	String	api_call_size
processors.RequestMetrics.labels	Labels for organizing metrics, such as HTTP method, URL, etc.	List	http_method, url, consumer_tag
processors.ResponseMetrics.metric_name	Name of the custom metric for responses.	String	rate_limit_remaining
processors.ResponseMetrics.metric_value	Value for the metric, such as the value of the X-Ratelimit-Remaining header.	String	"$[\"headers\"][\"X-Ratelimit-Remaining\"]"
processors.ResponseMetrics.metric_type	Type of the response metric (e.g., counter, gauge, histogram).	String	gauge