Flow Configuration Template
Flow Configuration
/etc/lunar-proxy/flows/flow.yaml
# Name of the flow for identification purposes
name: HARCollectorFlow
# Filter configuration to specify which requests the flow applies to
filter:
url: "api.com/resource/{id}" # Target URL pattern for the requests to be captured
# Processor configuration section
processors:
HARCollectorResponse:
processor: HARCollector # Specifies the processor used for HAR data collection
parameters:
- key: exporter_id
value: "file_exporter_01" # ID of the File Exporter defined in gateway_config.yaml
- key: transaction_max_size_bytes
value: 5000 # Maximum size limit for each HTTP transaction log (in bytes)
- key: obfuscate_enabled
value: true # Enables obfuscation of sensitive data in the logs
- key: obfuscate_exclusions
value:
- '$.request.query_param.id' # Exclude specific query parameters from obfuscation
- '$.request.body.user.name' # Exclude specific paths in the request body from obfuscation
- '$.response.headers["Retry-after"]'
metrics:
enabled: true # Enables metric collection for this processor
labels:
- flow_name
- processor_key
- http_method
- url
- status_code
# Flow definition section for request and response handling
flow:
request:
# Start of the request flow
- from:
stream:
name: globalStream # Use the global stream for capturing requests
at: start # Start capturing at the beginning of the stream
to:
stream:
name: globalStream # Return the request to the global stream
at: end # End point of the request flow
response:
# Start of the response flow
- from:
stream:
name: globalStream
at: start # Start of the response flow
to:
processor:
name: HARCollectorResponse # Capture the response using the HARCollector processor
- from:
processor:
name: HARCollectorResponse # Capture the response using the HARCollector processor
to:
stream:
name: globalStream # Send the processed response back to the global stream
at: end # End point of the response flow
Configuration Fields Explained
Field | Description | Example Value |
---|---|---|
exporter_id | Links to the File Exporter in gateway_config.yaml . | file_exporter_01 |
transaction_max_size_bytes | Sets the maximum size for each logged transaction (in bytes). | 5000 |
obfuscate_enabled | Enables or disables obfuscation of sensitive data. | true |
obfuscate_exclusions | Specifies fields to exclude from obfuscation. | See Obfuscation Exclusions |
metrics | Enables metric collection for the HAR Collector Processor. | enabled: true |
Obfuscation Exclusions
The HAR Metrics Collector Processor includes robust obfuscation capabilities to protect sensitive data. By default, if obfuscate_enabled
is set to true
, the processor will mask:
query_params
path_params
request_headers
response_headers
request_body_paths
response_body_paths
Example Obfuscation Configuration:
/etc/lunar-proxy/flows/flow.yaml
- key: obfuscate_enabled
value: true # Enables obfuscation of sensitive data in the logs
- key: obfuscate_exclusions
value:
- $.response.body.username
- $.request.headers.auth
In this configuration:
- Obfuscation is enabled for all data except for the specified exclusions.
- Excluded fields (e.g.,
id
,user.name
,Retry-After
) are not obfuscated, allowing specific data points to remain visible for diagnostics.
Obfuscation Details:
- Default Behavior: Without specifying
obfuscate_enabled
, obfuscation is disabled. - Hashing: Obfuscated values are replaced with consistent hashes, ensuring data privacy while maintaining traceability.
note
- If the
obfuscate
field is not specified, the plugin will not obfuscate any sensitive information. - If
obfuscate
is enabled, the plugin will obfuscate all query parameter values, path parameter values, request/response header values, request/response body values by default. - Obfuscation is done by replacing the original value with a hash of that value. This means that the obfuscated value will be the same for the same original value. For example, if the original value of a query parameter is
123
, the obfuscated value will always be the same hash of123
. - Query parameter names and path parameter names are not obfuscated. Only their values are obfuscated. The same goes for request/response header names and request/response body paths.
Gateway Configuration (gateway_config.yaml)
The gateway configuration defines the export settings for HAR data. The following should be added to your gateway_config.yaml
depending on your preferred exporter.
/etc/lunar-proxy/gateway_config.yaml
exporters:
file:
exporter_id: my_file_exporter
file_dir: "/var/log/lunar-proxy" # Directory where HAR logs will be stored
file_name: "transaction.log" # Naming pattern for log files
cloud:
exporter_id: my_s3_exporter
type: "s3" #gcp
bucket_name: "my-har-logs"
region: "us-east-1"
Configuration Fields Explained
Field | Description | Example Value |
---|---|---|
file.file_dir | Directory where HAR logs will be stored. | /var/log/lunar/har_logs |
file.file_name | Naming pattern for the log files. | har_exporter_{timestamp}.log |
cloud.type | Which cloud provider is being exported to. | s3 |
cloud.bucket_name | Name of bucket. | my-har-logs |
cloud.region | Region where bucket exists. | us-east-1 |
Troubleshooting
- Log File Not Created:
- Ensure
exporter_id
inflows.yaml
matches the File Exporter ID ingateway_config.yaml
. - Verify that the specified
file_dir
exists and has appropriate write permissions.
- Ensure
- Large Log Files:
- Use
transaction_max_size_bytes
to limit the size of each logged transaction. - Set
max_file_size
ingateway_config.yaml
to manage log file growth.
- Use
- Obfuscation Not Working:
- Check that
obfuscate_enabled
is set totrue
and verify the exclusions list. - Ensure that the fields specified for exclusion exist in the captured HTTP data.
- Check that