Fallback Flow

The Fallback Flow enables users to retry requests to an LLM provider and dynamically switch models based on user-defined criteria. These conditions are evaluated using the Filter Processor, which allows for granular control over when a fallback should be triggered. For example, if a request to GPT-4 times out or returns an error, the system can automatically retry the request using GPT-3.5.

---

Scenarios

1. LLM Rate Limit Reached: If a request returns with a specific response code, it can be retried and sent to a different LLM model instead of waiting for the specific LLM model to be available again.

---

Flow Components

• Filter Processor
• Retry Processor
• Transform API Call Processor

---

Flow Example

Here is an example of a fully configured Fallback Flow. In this case, if a request returns from OpenAI Model GPT-4.5 and a status code between the range 429-504, the request will be transformed and resent to OpenAI Model GPT-4.1.

/etc/lunar-proxy/flows/flow.yaml

name: OpenAIModelFallbackFlow
filter: 
  url: api.openai.com/*

processors: 
  DidRetryFilter: 
    processor: Filter
    parameters: 
      - key: headers
        value: 
          x-lunar-retry-attempt: "true"
  RetryProcessor: 
    processor: Retry
    parameters: 
      - key: attempts
        value: 1
  ShouldRetryFilter: 
    processor: Filter
    parameters: 
      - key: status_code_range
        value: "429-504"
      - key: expressions
        value: 
          - $.request[?(@.body.model == "gpt-4.5")]
  TransformProcessor: 
    processor: TransformAPICall
    parameters: 
      - key: set
        value: 
          $.request.body.model: gpt-4.1
flow: 
  request: 
    - from: 
        stream: 
          name: globalStream
          at: start
      to: 
        processor: 
          name: DidRetryFilter
    - from: 
        processor: 
          name: DidRetryFilter
          condition: miss
      to: 
        stream: 
          name: globalStream
          at: end
    - from: 
        processor: 
          name: DidRetryFilter
          condition: hit
      to: 
        processor: 
          name: TransformProcessor
    - from: 
        processor: 
          name: TransformProcessor
      to: 
        stream: 
          name: globalStream
          at: end
  response: 
    - from: 
        stream: 
          name: globalStream
          at: start
      to: 
        processor: 
          name: ShouldRetryFilter
    - from: 
        processor: 
          name: ShouldRetryFilter
          condition: miss
      to: 
        stream: 
          name: globalStream
          at: end
    - from: 
        processor: 
          name: ShouldRetryFilter
          condition: hit
      to: 
        processor: 
          name: RetryProcessor
    - from: 
        processor: 
          name: RetryProcessor
          condition: retry
      to: 
        stream: 
          name: globalStream
          at: end
    - from: 
        processor: 
          name: RetryProcessor
          condition: failed
      to: 
        stream: 
          name: globalStream
          at: end

---