Skip to main content
Version: Next

Priority Queue Flow

The Priority Queue Flow manages API requests based on their priority, ensuring high-priority traffic is handled first while maintaining smooth processing for lower-priority requests. Using the Queue processor, it controls the number of requests in the queue and assigns priorities via the x-lunar-consumer-tag header. Requests are delayed until their turn arrives or the time-to-live (TTL) expires. If the queue is full, or limits are exceeded, a 429 "Too Many Requests" response is generated.

This flow optimizes traffic load, prioritizes critical requests, and ensures efficient resource utilization.

Flow Diagram

Scenarios​

  1. Critical Request Prioritization: Ensure that high-priority requests, such as production traffic, are processed first in environments with mixed priorities.
  2. Preventing Overload with Queuing: Delay lower-priority requests in a queue to manage traffic without immediately generating "Too Many Requests" responses, unless the queue is full or TTL expires.
  3. Handling High-Traffic APIs: Manage APIs with mixed-priority requests, balancing both staging and production traffic efficiently.
  4. Fair Resource Allocation: Assign priority levels to different requests to balance resource usage and prevent low-priority traffic from impacting high-priority operations.
  5. Custom Error Handling: Configure responses for exceeding limits to ensure users are informed of delays or retries in a controlled manner.

Flow Configuration Template​

/etc/lunar-proxy/flows/flow.yaml
name: PriorityQueueFlow # The name of the flow

filter:
  url: <URLPattern> # Define the URL pattern for the filter, e.g., api.example.com/*
processors:
  Queue:
    processor: Queue # Queue processor for managing request flow
    parameters:
      - key: quota_id
        value: <QuotaID> # Specify the quota ID, e.g., MyQueue
      - key: ttl_seconds
        value: <TTLInSeconds> # Time to live for requests in the queue
      - key: queue_size
        value: <MaxQueueSize> # Maximum number of requests in the queue
      - key: priority_group_by_header
        value: <HeaderForPriorityGroup> # Optional: Header to determine request priority
      - key: priority_groups
        value: 
          <Group1>: <Priority1> # Optional: Priority groups with their respective priorities
          <Group2>: <Priority2>
  GenerateResponse:
    processor: GenerateResponse # Generate a response when the queue limit is reached
    parameters:
      - key: status
        value: 429 # HTTP status code for queue limit reached, e.g., 429 (Too Many Requests)
      - key: body
        value: "Queue Limit Exceeded. Please try again later." # Response body text
      - key: Content-Type
        value: text/plain # Content type for the response
flow:
  request:
    - from:
        stream:
          name: globalStream # The stream to start the request flow
          at: start # Start point
      to:
        processor:
          name: Queue # Process the request through the Queue processor
    - from:
        processor:
          name: Queue # After the Queue processor
          condition: blocked # If the request is blocked due to queue limit reached
      to:
        processor:
          name: GenerateResponse # Generate the 429 response
    - from:
        processor:
          name: Queue # After the Queue processor
          condition: allowed # If the request is allowed (below queue limit)
      to:
        stream:
          name: globalStream # Send the request to the global stream
          at: end # End point of the request flow
  response:
    - from:
        processor:
          name: GenerateResponse # In case of queue limit reached, send the response
      to:
        stream:
          name: globalStream # Send response back to the global stream
          at: end # End point of the response flow

Flow Example​

In this example API requests to api.website.com/* are queued with a limit of 10 requests per second. Requests are prioritized based on the x-lunar-consumer-tag header, where production traffic gets higher priority (1) over staging traffic (2). If the queue is full or a request is delayed beyond the defined TTL (12 seconds), the system returns a 429 HTTP status code with the message: "Too many requests. Please try again later."

/etc/lunar-proxy/flows/flow.yaml
name: PriorityQueueFlow

filter:
  url: api.website.com/*

processors:
  QueueLimiter:
    processor: Queue
    parameters:
      - key: quota_id
        value: MyQuota
      - key: ttl_seconds
        value: 12
      - key: queue_size
        value: 10
      - key: priority_group_by_header
        value: x-lunar-consumer-tag
      - key: priority_groups
        value:
          production: 1
          staging: 2
  GenerateResponseTooManyRequests:
    processor: GenerateResponse
    parameters:
      - key: status
        value: 429
      - key: body
        value: "Too many requests. Please try again later."
      - key: Content-Type
        value: text/plain
flow:
  request:
    - from:
        stream:
          name: globalStream
          at: start
      to:
        processor:
          name: QueueLimiter
    - from:
        processor:
          name: QueueLimiter
          condition: blocked
      to:
        processor:
          name: GenerateResponseTooManyRequests
    - from:
        processor:
          name: QueueLimiter
          condition: allowed
      to:
        stream:
          name: globalStream
          at: end
  response:
    - from:
        processor:
          name: GenerateResponseTooManyRequests
      to:
        stream:
          name: globalStream
          at: end
/etc/lunar-proxy/quotas/quota.yaml
quotas:
  - id: MyQuota
    filter:
      url: api.website.com/*
    strategy:
      fixed_window:
        static:
          max: 10
          interval: 1
          interval_unit: second

Flow Components​