Concurrency-Based Throttling

Overview

Concurrency Based Throttling enforces a policy of API throttling which involves setting a limit on the number of concurrent API requests. When applicable, at any given moment there should be no more than the configured max_concurrent_requests waiting for a response from API provider. Any exceeding request will return immediately with the configured response_status_code.

Configuration

/etc/lunar-proxy/policies.yaml

global:
  remedies:
    - name: Concurrency-Based Throttling Example
      enabled: true
      config:
        concurrency_based_throttling:
          max_concurrent_requests: <max_concurrent_requests>
          response_status_code: <response_status_code>

The following configuration options are available for this remediation plugin:

Parameter	Example Value	Possible Values	Description
max_concurrent_requests	20	Any positive integer	The maximal allowed concurrent requests
response_status_code	429	Any HTTP status code	The response status code to return for requests exceeding max_concurrent_requests

Examples

/etc/lunar-proxy/policies.yaml

endpoints:
  - url: api.com/resource/{id}
    method: GET
    remedies:
      - name: Concurrency-Based Throttling Example
        enabled: true
        config:
          concurrency_based_throttling:
            max_concurrent_requests: 50
            response_status_code: 429

In the above example, the plugin will enforce that at any given moment, 50 concurrent requests at most will be in transit (i.e. waiting for a response). If the limit is exceeded, the plugin will return a 429 HTTP status code.

Use Cases

• Substitute for missing rate limiting by API provider: In some scenarios, API providers do not implement actual rate limit mechanisms on their end; nonetheless, they would like API consumers to only consume at a certain concurrency level, whether agreed upon via SLAs or by mere goodwill. This plugin helps in solving this problem by avoiding more than a specified max_concurrent_requests.