Skip to main content
Version: Next

API Stream Filter

Overview​

A stream filter defines how traffic is segmented into sub-streams. Filters can be based on various parameters, such as:

  • url: API provider (URL) and specific endpoint (with wildcards or specific paths)
  • query_params: Query Parameters in the endpoint
  • path_params: Path Parameters in the endpoint
  • method: HTTP methods (e.g., POST, GET)
  • headers: Request header (key-value pairs)
  • response_headers: Response header (key-value pairs)
  • status_code: Status Code returned from API Provider
  • sample_percentage: A percentage of traffic to send to the Lunar.dev Flow
  • expressions: JSONPath expressions in the API Request or Response

Parameters that contain numerical values have the option to use the property operation to specify which logical operation will be enforced on that value. If it is not declared, the default behavior is an exact match (eq).

note

When configuring the Stream Filter, each parameter you define must match for a request to pass the filter—these parameters are combined using AND logic. However, each parameter has a different combination logic determining how the values declared will be combined.

Parameters​

url​

Required: True
Type: string

A single string of an API Provider and specific endpoint. This can include wildcards (*), specific paths, and paths with path parameters. URLs using Regex are not supported here.

Example:

url: api.httpbin.org/anything/{param_1}
tip

Wildcards can be used to match multiple subdomains of an API provider (e.g., *.httpbin.org) or to match any path suffix after a specific prefix (e.g., httpbin.org/anything/*).

query_params​

Required: False
Type: list of conditions
Combined Using: AND

A list of conditions used to match query parameters in the request URL. All conditions must match in order for this to work.

Example:

# api.httpbin.org/anything?query=1&user=admin
query_params:
  - key: query
    value: "1"
    operation: "eq" # or "exact"
  - key: user
    value: admin
    operation: "eq" # or "exact"

The default value for operation is eq, or exact, unless otherwise stated.

path_params​

Required: False
Type: list of conditions
Combined Using: AND

A list of conditions representing path parameters used in the endpoint. This allows you to filter requests based on specific values assigned to those path parameters. All conditions must match in order for this to work.

Example:

# api.httpbin.org/anything/{anything_id}/user/{user_id}
path_params:
  - key: anything_id
    value: [a-z0-9]{32}
    operation: "=~" # or "regex"
    - key: user_id
    value: 123
    operation: "eq"

The default value for operation is eq, or exact, unless otherwise stated.

method​

Required: False
Type: list of strings
Combined Using: OR

A HTTP method (e.g., POST, GET) sent as part of the request.

Example:

method:
  - GET

headers​

Required: False
Type: list of conditions
Combined Using: AND

A list of conditions used to match specific request headers and their values in the request. Each condition defines a header key, its expected value, and the matching operation.

Example:

headers:
  - key: user
    value: admin
    operation: "eq"
  - key: role
    value: production
    operation: "eq"

The default value for operation is eq, or exact, unless otherwise stated.

response_headers​

Required: False
Type: list of conditions
Combined Using: AND

A list of conditions used to match specific response headers and their values in the request. Each condition defines a header key, its expected value, and the matching operation.

Example:

response_headers:
  - key: user
    value: admin
    operation: "eq"
  - key: role
    value: production
    operation: "eq"

The default value for operation is eq, or exact, unless otherwise stated.

status_code​

Required: False
Type: list of integers
Combined Using: OR

A list of status codes returned from the API provider or generated by Lunar.dev.

Example:

status_code:
  - 200
  - 300

sample_percentage​

Required: False
Type: number

A single number between 1 and 100 that will determine which percentage of traffic will proceed to the Flow. This should be used when only a sample of the requests need to be sent to the Flow. This can be a whole number or a decimal to allow for specific sampling.

Example:

sample_percentage: 3.5

expressions​

Required: False
Type: list of conditions
Combined Using: AND

A list of conditions using JSONPath that can be used to used to extract specific values from the request or response url, header, method, or status for filtering purposes. When using expressions to filter API Streams, you cannot filter by elements from the body in order to do so please use the Filter Processor.

Example:

expressions:
  - key: $.request.method
    value: 'POST'
    operation: eq
  - key: $.request.method
    value: 'GET'
    operation: eq
  - key: $.response.status
    value: '200-300'

note

Expressions is an abstraction of all the other parameters that can be used to filter API Streams and therefore is a heavier parameter. Because of this it is recommended that you try to use the other parameters first if possible and only use expressions if necessary.

Template​

/etc/lunar-proxy/flows/flow.yaml
filter:
  url:
    - api.example.com/v1/*
  query_params:
    - key: query
      value: '1'
    - key: user
      value: admin
  path_params:
    - key: anything_id
      value: [a-z0-9]{32}
      operation: "regex" 
    - key: user_id
      value: 123
      operation: "eq" 
  methods: 
    - POST    
    - GET
  headers:
    - key: x-lunar-consumer-tag 
      value: premium 
    - key: x-lunar-consumer-tag 
      value: basic 
  response_headers:
    - key: x-lunar-consumer-tag 
      value: premium 
    - key: x-lunar-consumer-tag 
      value: basic 
  status_code:
    - 200
    - 300
  sample_percentage: 75  
  expressions:
    - key: $.request.method
      value: 'POST'
      operation: ==
    - key: $.response.status
      value: '200-300'
      operation: between

Use Case Example​

/etc/lunar-proxy/flows/flow.yaml
filter:
  url: api.example.com/v1/* 
  method: 
    - POST 
  headers:
    - key: x-lunar-consumer-tag
      value: premium 
    - key: x-lunar-consumer-tag 
      value: basic

In this setup, POST requests to api.example.com/v1/* are segmented based on whether they come from premium or basic users. This could be used in scenarios where you want to apply stricter rate limits to basic users but allow premium users to access the API more frequently.

For example, with this filter in place, you can implement separate rate-limiting rules for premium and basic users by associating different processors for each segment of traffic, enabling tailored traffic management.

Wildcards and Domain Matching​

When using wildcards in Stream Filters, you don't need to use regular expressions. Wildcards like *.api.example.com are accepted and make it easy to define patterns that match multiple subdomains or paths.

For instance, using a wildcard like:

filter:
  url: *.api.example.com

is perfectly valid and covers all subdomains such as dev.api.example.com or test.api.example.com, without the complexity of regex. This makes configuring filters more straightforward and user-friendly for scenarios where you need flexible matching across domains or URL patterns.

Another example: using a wildcard at the end of a path will match any additional path segments that follow it. For example:

filter:
  url: api.example.com/*

will match api.example.com/anything and api.example.com/something.

Supported Operations​

- 'eq' # equals
- 'neq' # not equals 
- 'gt' # greater than
- 'gte' # greater than or equal to
- 'lt' # less than
- 'lte' # less than or equal to
- 'exists' # exists
- 'not_exists' # does not exist
- 'regex' # regex

Troubleshooting​

  • If you would like to add an operation for 'between' you can use the operations gt and lt (or gte and lte)