Skip to main content
Version: Next

API Stream Filter

Overview​

A stream filter defines how traffic is segmented into sub-streams. Filters can be based on various parameters, such as:

  • url: API provider (URL) and specific endpoint (with wildcards or specific paths) (deprecated)
  • urls: List of API providers (URLs) and specific endpoints (with wildcards or specific paths)
  • query_params: Query Parameters in the endpoint
  • query_params_all_of: All of the Query Parameters in the endpoint
  • path_params: Path Parameters in the endpoint
  • path_params_all_of: All of Path Parameters in the endpoint
  • method: HTTP methods (e.g., POST, GET) (deprecated)
  • methods: List of HTTP methods (e.g., POST, GET)
  • headers: Request header (key-value pairs)
  • headers_all_of: All of the request header (key-value pairs)
  • response_headers: Response header (key-value pairs)
  • response_headers_all_of: All of the response header (key-value pairs)
  • status_code: Status Code returned from API Provider (deprecated)
  • status_codes: Status Code returned from API Provider
  • status_code_range: A range of Status Codes returned from API Provider (deprecated)
  • sample_percentage: A percentage of traffic to send to the Lunar.dev Flow
  • expressions: JSONPath expressions in the API Request or Response
  • expressions_all_of: All of the JSONPath expressions in the API Request or Response

Parameters that contain numerical values have the option to use the property operation to specify which logical operation will be enforced on that value. If it is not declared, the default behavior is an exact match (==).

note

When configuring the Stream Filter, each parameter you define must match for a request to pass the filter—these parameters are combined using AND logic. However, within each individual parameter, if you specify multiple values, they are combined using OR logic — only one of the values needs to match for that parameter.

The only parameters that are combined using AND logic are:

  • query_params_all_of
  • path_params_all_of
  • headers_all_of
  • response_headers_all_of
  • expressions_all_of

Parameters​

url (deprecated)​

Required: True
Type: string
A single string of an API Provider and specific endpoint. This can include wildcards (*), specific paths, and paths with path parameters. URLs using Regex are not supported here.

Example:

url: api.httpbin.org/anything/{param_1}
tip

Wildcards can be used to match multiple subdomains of an API provider (e.g., *.httpbin.org) or to match any path suffix after a specific prefix (e.g., httpbin.org/anything/*).

note

You can use only the parameter url or urls. They may not be used together in the same filter but the use of one of them is Required

urls​

Required: True
Type: list of strings
A list of strings of an API Provider and specific endpoints. This can include wildcards (*), specific paths, and paths with path parameters. URLs using Regex are not supported here.

Example:

# api.httpbin.org/anything/user_1/id_1
urls: 
  - api.httpbin.org/anything/{param_1}/{param_2}
tip

Wildcards can be used to match multiple subdomains of an API provider (e.g., *.httpbin.org) or to match any path suffix after a specific prefix (e.g., httpbin.org/anything/*).

query_params​

Required: False
Type: list of conditions

A list of conditions used to match query parameters in the request URL.

Example:

# api.httpbin.org/anything?query=1&user=admin
query_params:
  - key: query
    value: "1"
    operation: "==" # or "exact"
  - key: user
    value: admin
    operation: "==" # or "exact"

The default value for operation is ==, or exact, unless otherwise stated.

query_params_all_of​

Required: False
Type: list of conditions

A list of conditions used to match query parameters in the request URL. All of the conditions must be met in order to have a match.

Example:

# api.httpbin.org/anything?query=1&user=admin
query_params_all_of:
  - key: query
    value: "1"
    operation: "==" # or "exact"
  - key: user
    value: admin
    operation: "==" # or "exact"

The default value for operation is ==, or exact, unless otherwise stated.

path_params​

Required: False
Type: list of conditions

A list of conditions representing path parameters used in the endpoint. This allows you to filter requests based on specific values assigned to those path parameters.

Example:

# api.httpbin.org/anything/{anything_id}/user/{user_id}
path_params:
  - key: anything_id
    value: [a-z0-9]{32}
    operation: "=~" # or "regex"
    - key: user_id
    value: 123
    operation: "=="

The default value for operation is ==, or exact, unless otherwise stated.

path_params_all_of​

Required: False
Type: list of conditions

A list of conditions representing path parameters used in the endpoint. This allows you to filter requests based on specific values assigned to those path parameters. All of the conditions must be met in order to have a match.

Example:

# api.httpbin.org/anything/{anything_id}/user/{user_id}
path_params_all_of:
  - key: anything_id
    value: [a-z0-9]{32}
    operation: "=~" # or "regex"
    - key: user_id
    value: 123
    operation: "=="

The default value for operation is ==, or exact, unless otherwise stated.

method (deprecated)​

Required: False
Type: list of strings

A HTTP method (e.g., POST, GET) sent as part of the request.

Example:

method:
  - GET

methods​

Required: False
Type: list of strings

A list of HTTP methods (e.g., GET, POST, PUT, DELETE) to match against the request’s method. Useful when your filter should only apply to certain types of requests.

Example:

methods:
  - GET
  - POST

headers​

Required: False
Type: list of conditions

A list of conditions used to match specific request headers and their values in the request. Each condition defines a header key, its expected value, and the matching operation.

Example:

headers:
  - key: user
    value: admin
    operation: "=="
  - key: role
    value: production
    operation: "=="

The default value for operation is ==, or exact, unless otherwise stated.

headers_all_of​

Required: False
Type: list of conditions

A list of conditions used to match specific request headers and their values in the request. Each condition defines a header key, its expected value, and the matching operation. All of the conditions must be met in order to have a match.

Example:

headers:
  - key: user
    value: dev
    operation: "=="
  - key: env
    value: staging
    operation: "=="

The default value for operation is ==, or exact, unless otherwise stated.

response_headers​

Required: False
Type: list of conditions

A list of conditions used to match specific response headers and their values in the request. Each condition defines a header key, its expected value, and the matching operation.

Example:

response_headers:
  - key: user
    value: admin
    operation: "=="
  - key: role
    value: production
    operation: "=="

The default value for operation is ==, or exact, unless otherwise stated.

response_headers_all_of​

Required: False
Type: list of conditions

A list of conditions used to match specific response headers and their values in the request. Each condition defines a header key, its expected value, and the matching operation. All of the conditions must be met in order to have a match.

Example:

response_headers:
  - key: user
    value: dev
    operation: "=="
  - key: env
    value: staging
    operation: "=="

The default value for operation is ==, or exact, unless otherwise stated.

status_code (deprecated)​

Required: False
Type: list of integers

A list of status codes returned from the API provider or generated by Lunar.dev.

Example:

status_code:
  - 200
  - 300

status_code_range (deprecated)​

Required: False
Type: range of integers

An inclusive range of status codes returned from the API provider or generated by Lunar.dev.

Example:

status_code_range: '200-300'

status_codes​

Required: False
Type: list of integers

A list of status codes returned from the API provider or generated by Lunar.dev. If using a range of status codes, the codes are inclusive.

Example:

status_codes:
  - 200
  - '300-400'

sample_percentage​

Required: False
Type: number

A number between 1 and 100 that will determine which percentage of traffic will proceed to the Flow. This should be used when only a sample of the requests need to be sent to the Flow. This can be a whole number or a decimal to allow for specific sampling.

Example:

sample_percentage: 3.5

expressions​

Required: False
Type: list of conditions

A list of conditions using JSONPath that can be used to used to extract specific values from the request or response url, header, method, or status for filtering purposes. When using expressions to filter API Streams, you cannot filter by elements from the body in order to do so please use the Filter Processor.

Example:

expressions:
  - key: $.request.method
    value: 'POST'
    operation: ==
  - key: $.request.method
    value: 'GET'
    operation: ==
  - key: $.response.status
    value: '200-300'

note

Expressions is an abstraction of all the other parameters that can be used to filter API Streams and therefore is a heavier parameter. Because of this it is recommended that you try to use the other parameters first if possible and only use expression if necessary.

expressions_all_of​

Required: False
Type: list of conditions

A list of conditions using JSONPath that can be used to used to extract specific values from the request or response url, header, method, or status for filtering purposes. When using expressions to filter API Streams, you cannot filter by elements from the body in order to do so please use the Filter Processor.

Example:

expressions_all_of:
  - key: $.request.method
    value: 'POST'
    operation: ==
  - key: $.request.method
    value: 'GET'
    operation: ==
  - key: $.response.status
    value: '200-300'

note

Expressions_all_of is an abstraction of all the other parameters that can be used to filter API Streams and therefore is a heavier parameter. Because of this it is recommended that you try to use the other parameters first if possible and only use expression_all_of if necessary.

Template​

/etc/lunar-proxy/flows/flow.yaml
filter:
  urls:
    - api.example.com/v1/*
    - api.example.com/v2/*
  query_params:
    - key: query
      value: '1'
    - key: user
      value: admin
  query_params_all_of:
    - key: permissions
      value: admin
    - key: env
      value: production
  path_params:
    - key: anything_id
    value: [a-z0-9]{32}
    operation: "=~" 
    - key: user_id
    value: 123
    operation: "==" 
  path_params_all_of:
    - key: organization_id
    value: [a-z0-9]{32}
    operation: "=~"
    - key: property
    value: 123
    operation: "==" 
  methods: 
    - POST    
    - GET
  headers:
    - key: x-lunar-consumer-tag 
      value: premium 
    - key: x-lunar-consumer-tag 
      value: basic 
  headers_all_of:
    - key: user_tier 
      value: premium 
    - key: access_level
      value: basic 
  response_headers:
    - key: x-lunar-consumer-tag 
      value: premium 
    - key: x-lunar-consumer-tag 
      value: basic 
  response_headers_all_of:
    - key: user_tier 
      value: premium 
    - key: access_level
      value: basic 
  status_code:
    - 200
    - 300
  status_code_range:
    - '200-300'
  sample_percentage: 75  
  expressions:
    - key: $.request.method
      value: 'POST'
      operation: ==
    - key: $.response.status
      value: '200-300'
      operation: between
  expressions_all_of:
    - key: $.request.method
      value: 'POST'
      operation: ==
    - key: $.response.status
      value: '200-300'
      operation: between

Use Case Example​

/etc/lunar-proxy/flows/flow.yaml
filter:
  url: api.example.com/v1/* 
  method: 
    - POST 
  headers:
    - key: x-lunar-consumer-tag
      value: premium 
    - key: x-lunar-consumer-tag 
      value: basic

In this setup, POST requests to api.example.com/v1/* are segmented based on whether they come from premium or basic users. This could be used in scenarios where you want to apply stricter rate limits to basic users but allow premium users to access the API more frequently.

For example, with this filter in place, you can implement separate rate-limiting rules for premium and basic users by associating different processors for each segment of traffic, enabling tailored traffic management.

Wildcards and Domain Matching​

When using wildcards in Stream Filters, you don't need to use regular expressions. Wildcards like *.api.example.com are accepted and make it easy to define patterns that match multiple subdomains or paths.

For instance, using a wildcard like:

filter:
  url: *.api.example.com

is perfectly valid and covers all subdomains such as dev.api.example.com or test.api.example.com, without the complexity of regex. This makes configuring filters more straightforward and user-friendly for scenarios where you need flexible matching across domains or URL patterns.

Another example: using a wildcard at the end of a path will match any additional path segments that follow it. For example:

filter:
  url: api.example.com/*

will match api.example.com/anything and api.example.com/something.

Supported Operations​

- '==' # equals
- '=~' # regex
- '>' # greater than
- '>=' # greater than or equal to
- '<' # less than
- '<=' # less than or equal to
- 'exists' # exists
- 'between' # between two numbers
- '!=' # not equals

Troubleshooting​

  • When using the parameters query_params_all_of, path_params_all_of, headers_all_of, response_headers_all_of or expressions_all_of be sure not to define the same key as in query_params, path_params, headers, response_headers or expressions. Since the parameters are combined using AND logic, defining the same key twice will cause the a mismatch and the filter will not match.