Skip to main content
Version: 1.1.x

HAR Collector Flow

The HAR Collector Flow captures detailed HTTP transaction data, including headers, query parameters, and request/response bodies. It stores the data in a log file using a File Exporter, making it easy to trace API interactions for troubleshooting and auditing.

Flow Diagram

The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled is set to true, the processor will mask:

  • query_params
  • path_params
  • request_headers
  • response_headers
  • request_body_paths
  • response_body_paths
note
  • Without specifying obfuscate_enabled, obfuscation is disabled.
  • Obfuscated values are replaced with consistent hashes, ensuring data privacy while maintaining traceability.
  • If the obfuscate field is not specified, the plugin will not obfuscate any sensitive information.
  • If obfuscate is enabled, the plugin will obfuscate all query parameter values, path parameter values, request/response header values, request/response body values by default.
  • Obfuscation is done by replacing the original value with a hash of that value. This means that the obfuscated value will be the same for the same original value. For example, if the original value of a query parameter is 123, the obfuscated value will always be the same hash of 123.
  • Query parameter names and path parameter names are not obfuscated. Only their values are obfuscated. The same goes for request/response header names and request/response body paths.
  • If an address of where to write the logs is not specified in the gateway_config.yaml, the logs will be written to the fluent-bit folder in the your local Docker container.

Scenarios​

  1. Granular HTTP Data Capture: Record complete HTTP transactions for diagnostics.
  2. Data Obfuscation: Mask sensitive information with configurable obfuscation rules.
  3. Local Log Export: Store logs locally with customizable file paths and names.
  4. Metrics Integration: Monitor log collection and export using Prometheus-compatible metrics.

Flow Components​

Flow Example​

In this example, the flow captures HTTP transaction logs for a specific API endpoint and exports them using the configured File Exporter. Obfuscation is enabled for all data except for the specified exclusions (id, user.name, Retry-After), allowing specific data points to remain visible for diagnostics.

/etc/lunar-proxy/flows/flow.yaml
name: HARCollectorFlow

filter:
  url: "api.com/resource/{id}"
processors:
  HARCollectorResponse:
    processor: HARCollector 
    parameters:
      - key: exporter_id
        value: "file_exporter_01"
      - key: transaction_max_size_bytes
        value: 5000 
      - key: obfuscate_enabled
        value: true
      - key: obfuscate_exclusions
        value:
          - '$.request.query_param.id' # Exclude specific query parameters from obfuscation
          - '$.request.body.user.name' # Exclude specific paths in the request body from obfuscation
          - '$.response.headers["Retry-after"]'
    metrics:
      enabled: true  # Enables metric collection for this processor
      labels:
        - flow_name
        - processor_key
        - http_method
        - url
        - status_code
flow:
  request:
    - from:
        stream:
          name: globalStream 
          at: start  
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        stream:
          name: globalStream
          at: start 
      to:
        processor:
          name: HARCollectorResponse 
    - from:
        processor:
          name: HARCollectorResponse 
      to:
        stream:
          name: globalStream
          at: end

/etc/lunar-proxy/gateway_config.yaml
exporters:
  file:
    exporter_id: my_file_exporter
    file_dir: "/var/log/lunar-proxy"
    file_name: "transaction.log"

Troubleshooting​

  • Log File Not Created:
    • Ensure exporter_id in flows.yaml matches the File Exporter ID in gateway_config.yaml.
    • Verify that the specified file_dir exists and has appropriate write permissions.
  • Large Log Files:
    • Use transaction_max_size_bytes to limit the size of each logged transaction.
    • Set max_file_size in gateway_config.yaml to manage log file growth.
  • Obfuscation Not Working:
    • Check that obfuscate_enabled is set to true and verify the exclusions list.
    • Ensure that the fields specified for exclusion exist in the captured HTTP data.