HAR Collector Flow

The HAR Collector Flow captures detailed HTTP transaction data, including headers, query parameters, and request/response bodies. It stores the data in a log file using a File Exporter, making it easy to trace API interactions for troubleshooting and auditing.

The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled is set to true, the processor will mask:

• query_params
• path_params
• request_headers
• response_headers
• request_body_paths
• response_body_paths

note

• Without specifying obfuscate_enabled, obfuscation is disabled.
• Obfuscated values are replaced with consistent hashes, ensuring data privacy while maintaining traceability.
• If the obfuscate field is not specified, the plugin will not obfuscate any sensitive information.
• If obfuscate is enabled, the plugin will obfuscate all query parameter values, path parameter values, request/response header values, request/response body values by default.
• Obfuscation is done by replacing the original value with a hash of that value. This means that the obfuscated value will be the same for the same original value. For example, if the original value of a query parameter is 123, the obfuscated value will always be the same hash of 123.
• Query parameter names and path parameter names are not obfuscated. Only their values are obfuscated. The same goes for request/response header names and request/response body paths.

---

Scenarios

1. Granular HTTP Data Capture: Record complete HTTP transactions for diagnostics.
2. Data Obfuscation: Mask sensitive information with configurable obfuscation rules.
3. Local Log Export: Store logs locally with customizable file paths and names.
4. Metrics Integration: Monitor log collection and export using Prometheus-compatible metrics.

---

Flow Configuration Template

/etc/lunar-proxy/flows/flow.yaml

name: HARCollectorFlow

filter:
  url: "api.com/resource/{id}"  # Target URL pattern for the requests to be captured
processors:
  HARCollectorResponse:
    processor: HARCollector  # Specifies the processor used for HAR data collection
    parameters:
      - key: exporter_id
        value: "file_exporter_01"  # ID of the File Exporter defined in gateway_config.yaml
      - key: transaction_max_size_bytes
        value: 5000  # Maximum size limit for each HTTP transaction log (in bytes)
      - key: obfuscate_enabled
        value: true  # Enables obfuscation of sensitive data in the logs
      - key: obfuscate_exclusions
        value:
          - '$.request.query_param.id' # Exclude specific query parameters from obfuscation
          - '$.request.body.user.name' # Exclude specific paths in the request body from obfuscation
          - '$.response.headers["Retry-after"]' 
    metrics:
      enabled: true  # Enables metric collection for this processor
      labels:
        - flow_name
        - processor_key
        - http_method
        - url
        - status_code  
flow:
  request:
    - from:
        stream:
          name: globalStream  # Use the global stream for capturing requests
          at: start  # Start capturing at the beginning of the stream
      to:
        stream:
          name: globalStream  # Return the request to the global stream
          at: end  # End point of the request flow   
  response:
    - from:
        stream:
          name: globalStream
          at: start # Start of the response flow
      to:
        processor:
          name: HARCollectorResponse # Capture the response using the HARCollector processor
    - from:
        processor:
          name: HARCollectorResponse  # Capture the response using the HARCollector processor
      to:
        stream:
          name: globalStream  # Send the processed response back to the global stream
          at: end  # End point of the response flow

The gateway configuration defines the export settings for HAR data. The following should be added to your gateway_config.yaml depending on your preferred exporter.

/etc/lunar-proxy/gateway_config.yaml

exporters:
  file:
    exporter_id: my_file_exporter
    file_dir: "/var/log/lunar-proxy" # Directory where HAR logs will be stored
    file_name: "transaction.log" # Naming pattern for log files

  cloud:
    exporter_id: my_s3_exporter
    type: "s3" #gcp
    bucket_name: "my-har-logs"
    region: "us-east-1"

---

Flow Example

In this example, the flow captures HTTP transaction logs for a specific API endpoint and exports them using the configured File Exporter. Obfuscation is enabled for all data except for the specified exclusions (id, user.name, Retry-After), allowing specific data points to remain visible for diagnostics.

/etc/lunar-proxy/flows/flow.yaml

name: HARCollectorFlow

filter:
  url: "api.com/resource/{id}"
processors:
  HARCollectorResponse:
    processor: HARCollector 
    parameters:
      - key: exporter_id
        value: "file_exporter_01"
      - key: transaction_max_size_bytes
        value: 5000 
      - key: obfuscate_enabled
        value: true
      - key: obfuscate_exclusions
        value:
          - '$.request.query_param.id' # Exclude specific query parameters from obfuscation
          - '$.request.body.user.name' # Exclude specific paths in the request body from obfuscation
          - '$.response.headers["Retry-after"]'
    metrics:
      enabled: true  # Enables metric collection for this processor
      labels:
        - flow_name
        - processor_key
        - http_method
        - url
        - status_code
flow:
  request:
    - from:
        stream:
          name: globalStream 
          at: start  
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        stream:
          name: globalStream
          at: start 
      to:
        processor:
          name: HARCollectorResponse 
    - from:
        processor:
          name: HARCollectorResponse 
      to:
        stream:
          name: globalStream
          at: end

/etc/lunar-proxy/gateway_config.yaml

exporters:
  file:
    exporter_id: my_file_exporter
    file_dir: "/var/log/lunar-proxy"
    file_name: "transaction.log"

---

Flow Components

• HARCollector Processor
• gateway_config.yaml

---

Troubleshooting

• Log File Not Created:
  • Ensure exporter_id in flows.yaml matches the File Exporter ID in gateway_config.yaml.
  • Verify that the specified file_dir exists and has appropriate write permissions.
• Large Log Files:
  • Use transaction_max_size_bytes to limit the size of each logged transaction.
  • Set max_file_size in gateway_config.yaml to manage log file growth.
• Obfuscation Not Working:
  • Check that obfuscate_enabled is set to true and verify the exclusions list.
  • Ensure that the fields specified for exclusion exist in the captured HTTP data.

---