Flow Configuration Template

Flow Configuration

/etc/lunar-proxy/flows/flow.yaml

# Name of the flow for identification purposes
name: HARCollectorFlow

# Filter configuration to specify which requests the flow applies to
filter:
  url: "api.com/resource/{id}"  # Target URL pattern for the requests to be captured

# Processor configuration section
processors:
  HARCollectorResponse:
    processor: HARCollector  # Specifies the processor used for HAR data collection
    parameters:
      - key: exporter_id
        value: "file_exporter_01"  # ID of the File Exporter defined in gateway_config.yaml
      - key: transaction_max_size_bytes
        value: 5000  # Maximum size limit for each HTTP transaction log (in bytes)
      - key: obfuscate_enabled
        value: true  # Enables obfuscation of sensitive data in the logs
      - key: obfuscate_exclusions
        value:
          - '$.request.query_param.id' # Exclude specific query parameters from obfuscation
          - '$.request.body.user.name' # Exclude specific paths in the request body from obfuscation
          - '$.response.headers["Retry-after"]' 
    metrics:
      enabled: true  # Enables metric collection for this processor
      labels:
        - flow_name
        - processor_key
        - http_method
        - url
        - status_code
          
# Flow definition section for request and response handling
flow:
  request:
    # Start of the request flow
    - from:
        stream:
          name: globalStream  # Use the global stream for capturing requests
          at: start  # Start capturing at the beginning of the stream
      to:
        stream:
          name: globalStream  # Return the request to the global stream
          at: end  # End point of the request flow   

  response:
    # Start of the response flow
    - from:
        stream:
          name: globalStream
          at: start # Start of the response flow
      to:
        processor:
          name: HARCollectorResponse # Capture the response using the HARCollector processor
    - from:
        processor:
          name: HARCollectorResponse  # Capture the response using the HARCollector processor
      to:
        stream:
          name: globalStream  # Send the processed response back to the global stream
          at: end  # End point of the response flow

Configuration Fields Explained

Field	Description	Example Value
exporter_id	Links to the File Exporter in gateway_config.yaml.	file_exporter_01
transaction_max_size_bytes	Sets the maximum size for each logged transaction (in bytes).	5000
obfuscate_enabled	Enables or disables obfuscation of sensitive data.	true
obfuscate_exclusions	Specifies fields to exclude from obfuscation.	See Obfuscation Exclusions
metrics	Enables metric collection for the HAR Collector Processor.	enabled: true

Obfuscation Exclusions

The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled is set to true, the processor will mask:

• query_params
• path_params
• request_headers
• response_headers
• request_body_paths
• response_body_paths

Example Obfuscation Configuration:

/etc/lunar-proxy/flows/flow.yaml

      - key: obfuscate_enabled
        value: true  # Enables obfuscation of sensitive data in the logs
      - key: obfuscate_exclusions
        value:
          - $.response.body.username
          - $.request.headers.auth 

In this configuration:

• Obfuscation is enabled for all data except for the specified exclusions.
• Excluded fields (e.g., id, user.name, Retry-After) are not obfuscated, allowing specific data points to remain visible for diagnostics.

Obfuscation Details:

• Default Behavior: Without specifying obfuscate_enabled, obfuscation is disabled.
• Hashing: Obfuscated values are replaced with consistent hashes, ensuring data privacy while maintaining traceability.

note

• If the obfuscate field is not specified, the plugin will not obfuscate any sensitive information.
• If obfuscate is enabled, the plugin will obfuscate all query parameter values, path parameter values, request/response header values, request/response body values by default.
• Obfuscation is done by replacing the original value with a hash of that value. This means that the obfuscated value will be the same for the same original value. For example, if the original value of a query parameter is 123, the obfuscated value will always be the same hash of 123.
• Query parameter names and path parameter names are not obfuscated. Only their values are obfuscated. The same goes for request/response header names and request/response body paths.

Gateway Configuration (gateway_config.yaml)

The gateway configuration defines the export settings for HAR data. The following should be added to your gateway_config.yaml depending on your preferred exporter.

/etc/lunar-proxy/gateway_config.yaml

exporters:
  file:
    exporter_id: my_file_exporter
    file_dir: "/var/log/lunar-proxy" # Directory where HAR logs will be stored
    file_name: "transaction.log" # Naming pattern for log files

  cloud:
    exporter_id: my_s3_exporter
    type: "s3" #gcp
    bucket_name: "my-har-logs"
    region: "us-east-1"

Configuration Fields Explained

Field	Description	Example Value
file.file_dir	Directory where HAR logs will be stored.	/var/log/lunar/har_logs
file.file_name	Naming pattern for the log files.	har_exporter_{timestamp}.log
cloud.type	Which cloud provider is being exported to.	s3
cloud.bucket_name	Name of bucket.	my-har-logs
cloud.region	Region where bucket exists.	us-east-1

Troubleshooting

• Log File Not Created:
  • Ensure exporter_id in flows.yaml matches the File Exporter ID in gateway_config.yaml.
  • Verify that the specified file_dir exists and has appropriate write permissions.
• Large Log Files:
  • Use transaction_max_size_bytes to limit the size of each logged transaction.
  • Set max_file_size in gateway_config.yaml to manage log file growth.
• Obfuscation Not Working:
  • Check that obfuscate_enabled is set to true and verify the exclusions list.
  • Ensure that the fields specified for exclusion exist in the captured HTTP data.