Skip to main content
Version: Next

HAR Collector Flow

The HAR Collector Flow captures detailed HTTP transaction data, including headers, query parameters, and request/response bodies. It stores the data in a log file using a File Exporter, making it easy to trace API interactions for troubleshooting and auditing.

Flow Diagram

The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled is set to true, the processor will mask:

  • query_params
  • path_params
  • request_headers
  • response_headers
  • request_body_paths
  • response_body_paths

The HAR Metrics Collector Processor only writes structured logs, it does not create a .har file by itself. To generate a valid .har file, run the har-convert.js script with the following arguments:

HAR Convert Script
node har-convert.js <input_log_path> <output_har_path>

The input_log_path is defined in your gateway_config.yaml. Running this script will generate a .har file that can be read in your preferred browser.

note
  • Obfuscation is disabled by default. To enable it, set obfuscate_enabled: true
  • When enabled, the processor obfuscates values, not names, in query/path parameters, request/response headers, and bodies.
  • Obfuscation replaces values with consistent hashes to protect privacy while preserving traceability (e.g., "123" always hashes to the same value).
  • If no log output path is set in gateway_config.yaml, logs are written to the fluent-bit folder in your local Docker container.

Scenarios​

  1. Granular HTTP Data Capture: Record complete HTTP transactions for diagnostics.
  2. Data Obfuscation: Mask sensitive information with configurable obfuscation rules.
  3. Local Log Export: Store logs locally with customizable file paths and names.
  4. Metrics Integration: Monitor log collection and export using Prometheus-compatible metrics.

Flow Components​

Flow Example​

In this example, the flow captures HTTP transaction logs for a specific API endpoint and exports them using the configured File Exporter. Obfuscation is enabled for all data except for the specified exclusions (id, user.name, Retry-After), allowing specific data points to remain visible for diagnostics.

/etc/lunar-proxy/flows/flow.yaml
name: HARCollectorFlow

filter:
  url: "api.com/resource/{id}"
processors:
  HARCollectorResponse:
    processor: HARCollector 
    parameters:
      - key: exporter_id
        value: "file_exporter_01"
      - key: transaction_max_size_bytes
        value: 5000 
      - key: obfuscate_enabled
        value: true
      - key: obfuscate_exclusions
        value:
          - '$.request.query_param.id' # Exclude specific query parameters from obfuscation
          - '$.request.body.user.name' # Exclude specific paths in the request body from obfuscation
          - '$.response.headers["Retry-after"]'
    metrics:
      enabled: true  # Enables metric collection for this processor
      labels:
        - flow_name
        - processor_key
        - http_method
        - url
        - status_code
flow:
  request:
    - from:
        stream:
          name: globalStream 
          at: start  
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        stream:
          name: globalStream
          at: start 
      to:
        processor:
          name: HARCollectorResponse 
    - from:
        processor:
          name: HARCollectorResponse 
      to:
        stream:
          name: globalStream
          at: end

/etc/lunar-proxy/gateway_config.yaml
exporters:
  file:
    exporter_id: my_file_exporter
    file_dir: "/var/log/lunar-proxy"
    file_name: "transaction.log"
har-convert.js
node har-convert.js '/Downloads/transaction.log' '/Downloads/transaction.har'

Troubleshooting​

  • Log File Not Created:
    • Ensure exporter_id in flows.yaml matches the File Exporter ID in gateway_config.yaml.
    • Verify that the specified file_dir exists and has appropriate write permissions.
  • Large Log Files:
    • Use transaction_max_size_bytes to limit the size of each logged transaction.
    • Set max_file_size in gateway_config.yaml to manage log file growth.
  • Obfuscation Not Working:
    • Check that obfuscate_enabled is set to true and verify the exclusions list.
    • Ensure that the fields specified for exclusion exist in the captured HTTP data.
  • HAR File Not Generated:
    • Remember that the flow does not automatically generate a .har file
    • Run the har-convert.js script manually, providing the log file path as input and the destination for the output HAR file.