Skip to main content
Version: 0.8.x

Concurrency-Based Throttling

Overview

Concurrency Based Throttling enforces a policy of API throttling which involves setting a limit on the number of concurrent API requests. When applicable, at any given moment there should be no more than the configured max_concurrent_requests waiting for a response from API provider. Any exceeding request will return immediately with the configured response_status_code.

Configuration

/etc/lunar-proxy/policies.yaml
global:
remedies:
- name: Concurrency-Based Throttling Example
enabled: true
config:
concurrency_based_throttling:
max_concurrent_requests: <max_concurrent_requests>
response_status_code: <response_status_code>

The following configuration options are available for this remediation plugin:

ParameterExample ValuePossible ValuesDescription
max_concurrent_requests20Any positive integerThe maximal allowed concurrent requests
response_status_code429Any HTTP status codeThe response status code to return for requests exceeding max_concurrent_requests

Examples

/etc/lunar-proxy/policies.yaml
endpoints:
- url: api.com/resource/{id}
method: GET
remedies:
- name: Concurrency-Based Throttling Example
enabled: true
config:
concurrency_based_throttling:
max_concurrent_requests: 50
response_status_code: 429

In the above example, the plugin will enforce that at any given moment, 50 concurrent requests at most will be in transit (i.e. waiting for a response). If the limit is exceeded, the plugin will return a 429 HTTP status code.

Use Cases

  • Substitute for missing rate limiting by API provider: In some scenarios, API providers do not implement actual rate limit mechanisms on their end; nonetheless, they would like API consumers to only consume at a certain concurrency level, whether agreed upon via SLAs or by mere goodwill. This plugin helps in solving this problem by avoiding more than a specified max_concurrent_requests.
Click me for guidance 😀