Strategy-Based Queue
Overview
Strategy Based Queue introduces a delayed, priority-based queuing system. This plugin allows you to define a consumption rate by configuring window size and quota, as well as priorities for different groups, ensuring that higher-priority requests are processed first. By using this plugin, requests will be delayed until their turn to proceed has arrived, or until the defined TTL is met.
Configuration
global:
remedies:
- name: Prioritized Bursts Rate Limit
enabled: true
config:
strategy_based_queue:
allowed_request_count: <allowed_request_count>
window_size_in_seconds: <window_size_in_seconds>
ttl_seconds: <ttl_seconds>
queue_size: <size_of_max_queue_connections_in_number>
response_status_code: <response_status_code>
prioritization:
group_by:
header_name: <header_name>
groups:
<group_name1>:
priority: <priority1>
<group_name2>:
priority: <priority2>
The following configuration options are available for this plugin:
| Parameter | Example Value | Possible Values | Description |
|---|---|---|---|
| allowed_request_count | 1 | Any positive integer | The number of requests allowed within each window. |
| window_size_in_seconds | 1 | Any positive integer | The window size in seconds. |
| ttl_seconds | 12 | Any positive integer | Time-to-live for a request in the queue in seconds. |
| queue_size | 100 | Any positive integer | The maximum amount of requests the Proxy can hold in its queue. |
| response_status_code | 429 | Any HTTP status code | The HTTP status code returned when the TTL is exceeded. |
| prioritization.group_by.header_name | X-Group | Any header name | The header name to extract priority group from. |
| prioritization.groups.group_name | production | Any group name | The group name for prioritization. |
| prioritization.groups.*.priority | 1 | Any integer above 0 (including 0) | The priority level assigned to the group. Lower numbers indicate higher priority. |
Consider the queue_size setting carefully.
This value determines the number of open connections allowed between your client application and Lunar Proxy. Setting it too high can lead to memory exhaustion due to a large number of open sockets.
The prioritization section is optional.
Enqueued requests within the same priority are served by the order upon which they arrived. Enqueued requests of more urgent priority will always be served before other enqueued requests.
It is possible to define the same priority for more than one group.
Examples
endpoints:
- url: api.com/resource/{id}
method: GET
remedies:
- name: Prioritized Bursts Rate Limit
enabled: true
config:
strategy_based_queue:
allowed_request_count: 1
window_size_in_seconds: 1
ttl_seconds: 12
queue_size: 10
response_status_code: 429
prioritization:
group_by:
header_name: X-Group
groups:
production:
priority: 1
staging:
priority: 2
In this example, requests with the X-Group: production header will be given higher priority over those with X-Group: staging in the queue.
Use Cases
-
Avoiding Hitting Rate Limits: By delaying requests until there are available slots for them, Too Many Requests responses can be avoided.
-
Prioritizing Critical Requests: In scenarios where certain API requests are more critical than others (e.g., production vs. staging), this plugin allows prioritizing these critical requests to ensure business goals are met.
This plugin offers a refined level of control over API request management, catering to the needs of complex systems that require delayed, prioritized handling of API calls.