Flow Configuration Template

Flow Configuration

/etc/lunar-proxy/flows/flow.yaml

# Name of the flow for identification purposes
name: HARMetricsExporterFlow

# Filter configuration to specify which requests the flow applies to
filter:
  url: "api.com/resource/{id}"  # Target URL pattern for the requests to be captured

# Processor configuration section
processors:
  HARCollectorRequest:
    processor: HARMetricsCollector  # Specifies the processor used for HAR data collection
    parameters:
      - key: exporter_id
        value: "file_exporter_01"  # ID of the File Exporter defined in gateway_config.yaml
      - key: transaction_max_size
        value: "5000"  # Maximum size limit for each HTTP transaction log (in bytes)
      - key: obfuscate_enabled
        value: "true"  # Enables obfuscation of sensitive data in the logs
      - key: obfuscate_exclusions
        value:
          - $.request.query_param.id # Exclude specific query parameters from obfuscation
          - $.request.body.user.name # Exclude specific paths in the request body from obfuscation

  HARCollectorResponse:
    processor: HARMetricsCollector  # Specifies the processor used for HAR data collection
    parameters:
      - key: exporter_id
        value: "file_exporter_01"  # ID of the File Exporter defined in gateway_config.yaml
      - key: transaction_max_size
        value: "5000"  # Maximum size limit for each HTTP transaction log (in bytes)
      - key: obfuscate_enabled
        value: "true"  # Enables obfuscation of sensitive data in the logs
      - key: obfuscate_exclusions
        value:
          - $.response.headers["Retry-after"] 
          
# Flow definition section for request and response handling
flow:
  request:
    # Start of the request flow
    - from:
        stream:
          name: globalStream  # Use the global stream for capturing requests
          at: start  # Start capturing at the beginning of the stream
      to:
        processor:
          name: HARCollectorRequest  # Route the request to the HARCollector processor

    # End of the request flow, returning to the global stream
    - from:
        processor:
          name: HARCollectorRequest  # After the HARCollector processor finishes
      to:
        stream:
          name: globalStream  # Return the request to the global stream
          at: end  # End point of the request flow

  response:
    # Start of the response flow
    - from:
        stream:
          name: globalStream
          at: start # Start of the response flow
      to:
        processor:
          name: HARCollectorResponse # Capture the response using the HARCollector processor
    - from:
        processor:
          name: HARCollectorResponse  # Capture the response using the HARCollector processor
      to:
        stream:
          name: globalStream  # Send the processed response back to the global stream
          at: end  # End point of the response flow

Configuration Fields Explained

Field	Description	Example Value
exporter_id	Links to the File Exporter in gateway_config.yaml.	file_exporter_01
transaction_max_size	Sets the maximum size for each logged transaction (in bytes).	5000
obfuscate_enabled	Enables or disables obfuscation of sensitive data.	true
obfuscate_exclusions	Specifies fields to exclude from obfuscation.	See Obfuscation Exclusions
metrics	Enables metric collection for the HAR Collector Processor.	enabled: true

Obfuscation Exclusions

The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled is set to true, the processor will mask:

• query_params
• path_params
• request_headers
• response_headers
• request_body_paths
• response_body_paths

Example Obfuscation Configuration:

/etc/lunar-proxy/flows/flow.yaml

      - key: obfuscate_enabled
        value: "true"  # Enables obfuscation of sensitive data in the logs
      - key: obfuscate_exclusions
        value:
          - $.response.body.username
          - $.request.headers.auth 

In this configuration:

• Obfuscation is enabled for all data except for the specified exclusions.
• Excluded fields (e.g., id, user.name, Retry-After) are not obfuscated, allowing specific data points to remain visible for diagnostics.

Obfuscation Details:

• Default Behavior: Without specifying obfuscate_enabled, obfuscation is disabled.
• Hashing: Obfuscated values are replaced with consistent hashes, ensuring data privacy while maintaining traceability.

note

• If the obfuscate field is not specified, the plugin will not obfuscate any sensitive information.
• If obfuscate is enabled, the plugin will obfuscate all query parameter values, path parameter values, request/response header values, request/response body values by default.
• Obfuscation is done by replacing the original value with a hash of that value. This means that the obfuscated value will be the same for the same original value. For example, if the original value of a query parameter is 123, the obfuscated value will always be the same hash of 123.
• Query parameter names and path parameter names are not obfuscated. Only their values are obfuscated. The same goes for request/response header names and request/response body paths.

Gateway Configuration (gateway_config.yaml)

The gateway configuration defines the file export settings for HAR data.

/etc/lunar-proxy/gateway_config.yaml

file_exporters:
  - id: file_exporter_01
    file_dir: "/var/log/lunar/har_logs"  # Directory where HAR logs will be stored
    file_name: "har_exporter_{timestamp}.log"  # Naming pattern for log files
    max_file_size: 10485760  # Maximum file size in bytes (10 MB)

Configuration Fields Explained

Field	Description	Example Value
file_dir	Directory where HAR logs will be stored.	/var/log/lunar/har_logs
file_name	Naming pattern for the log files.	har_exporter_{timestamp}.log
max_file_size	Maximum size of a single log file in bytes.	10485760

Troubleshooting

• Log File Not Created:
  • Ensure exporter_id in flows.yaml matches the File Exporter ID in gateway_config.yaml.
  • Verify that the specified file_dir exists and has appropriate write permissions.
• Large Log Files:
  • Use transaction_max_size to limit the size of each logged transaction.
  • Set max_file_size in gateway_config.yaml to manage log file growth.
• Obfuscation Not Working:
  • Check that obfuscate_enabled is set to true and verify the exclusions list.
  • Ensure that the fields specified for exclusion exist in the captured HTTP data.