Flow Configuration Template

Flow Configuration

/etc/lunar-proxy/flows/flow.yaml

# Name of the flow for identification purposes
name: HARMetricsExporterFlow

# Filter configuration to specify which requests the flow applies to
filter:
  url: "api.com/resource/{id}"  # Target URL pattern for the requests to be captured

# Processor configuration section
processors:
  HARCollector:
    processor: HARMetricsCollector  # Specifies the processor used for HAR data collection
    parameters:
      - key: exporter_id
        value: file_exporter_01  # ID of the File Exporter defined in gateway_config.yaml
      - key: transaction_max_size
        value: 5000  # Maximum size limit for each HTTP transaction log (in bytes)
      - key: obfuscate_enabled
        value: true  # Enables obfuscation of sensitive data in the logs
      - key: obfuscate_exclusions
        value:
          query_params: ["id"]  # Exclude specific query parameters from obfuscation
          request_body_paths: ["user.name"]  # Exclude specific paths in the request body from obfuscation
          response_headers: ["Retry-After"]  # Exclude specific response headers from obfuscation

    # Metrics configuration for the processor
    metrics:
      enabled: true  # Enables metric collection for this processor
      labels:
        - flow_name      # Include the flow name as a label for metrics
        - processor_key  # Include the processor key as a label
        - http_method    # Include the HTTP method (GET, POST, etc.)
        - url            # Include the URL of the request
        - status_code    # Include the HTTP status code of the response

# Flow definition section for request and response handling
flow:
  request:
    # Start of the request flow
    - from:
        stream:
          name: globalStream  # Use the global stream for capturing requests
          at: start  # Start capturing at the beginning of the stream
      to:
        processor:
          name: HARCollector  # Route the request to the HARCollector processor

    # End of the request flow, returning to the global stream
    - from:
        processor:
          name: HARCollector  # After the HARCollector processor finishes
      to:
        stream:
          name: globalStream  # Return the request to the global stream
          at: end  # End point of the request flow

  response:
    # Start of the response flow
    - from:
        processor:
          name: HARCollector  # Capture the response using the HARCollector processor
      to:
        stream:
          name: globalStream  # Send the processed response back to the global stream
          at: end  # End point of the response flow

Configuration Fields Explained

Field	Description	Example Value
exporter_id	Links to the File Exporter in gateway_config.yaml.	file_exporter_01
transaction_max_size	Sets the maximum size for each logged transaction (in bytes).	5000
obfuscate_enabled	Enables or disables obfuscation of sensitive data.	true
obfuscate_exclusions	Specifies fields to exclude from obfuscation.	See Obfuscation Exclusions
metrics	Enables metric collection for the HAR Collector Processor.	enabled: true

Obfuscation Exclusions

The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled is set to true, the processor will mask:

• query_params
• path_params
• request_headers
• response_headers
• request_body_paths
• response_body_paths

Example Obfuscation Configuration:

/etc/lunar-proxy/flows/flow.yaml

obfuscate_enabled: true
obfuscate_exclusions:
  query_params:
    - "id"
  request_body_paths:
    - "user.name"
  response_headers:
    - "Retry-After"

In this configuration:

• Obfuscation is enabled for all data except for the specified exclusions.
• Excluded fields (e.g., id, user.name, Retry-After) are not obfuscated, allowing specific data points to remain visible for diagnostics.

Obfuscation Details:

• Default Behavior: Without specifying obfuscate_enabled, obfuscation is disabled.
• Hashing: Obfuscated values are replaced with consistent hashes, ensuring data privacy while maintaining traceability.

note

• If the obfuscate field is not specified, the plugin will not obfuscate any sensitive information.
• If obfuscate is enabled, the plugin will obfuscate all query parameter values, path parameter values, request/response header values, request/response body values by default.
• Obfuscation is done by replacing the original value with a hash of that value. This means that the obfuscated value will be the same for the same original value. For example, if the original value of a query parameter is 123, the obfuscated value will always be the same hash of 123.
• Query parameter names and path parameter names are not obfuscated. Only their values are obfuscated. The same goes for request/response header names and request/response body paths.

Troubleshooting

• Log File Not Created:
  • Ensure exporter_id in flows.yaml matches the File Exporter ID in gateway_config.yaml.
  • Verify that the specified file_dir exists and has appropriate write permissions.
• Large Log Files:
  • Use transaction_max_size to limit the size of each logged transaction.
  • Set max_file_size in gateway_config.yaml to manage log file growth.
• Obfuscation Not Working:
  • Check that obfuscate_enabled is set to true and verify the exclusions list.
  • Ensure that the fields specified for exclusion exist in the captured HTTP data.