​
Overview

Sometimes the model providers (OpenAI, Anthropic, etc.) you are using in your application become overloaded or you hit rate limits. Unless you have fallbacks and retry mechanisms in place, this may cause your application to fail too.

This cookbook shows how to add a global fallback that retries each request with an equivalent model on a different provider if the request fails. In this particular example, we’ll be switching to Claude 3.7 if GPT-4o fails.

​
Prerequisites

  • You’ve configured at least 2 model providers on your Waystone project.
  • You’ve created an API key for your Waystone project.
  • You know how to make requests to the Waystone API. See the quickstart if not!

​
Step 1 - Create a rule to retry with a fallback

In Waystone, rules are atomic components consisting of a condition and an action. Rules do not inherently apply to any users, but do so by being added to any number of rulesets.

  1. Head to your project on Waystone and click the rules button on the nav bar.
  2. Create a new rule, giving it an appropriate name and description.
  3. For the condition, select the Model condition and pick GPT-4o. This will apply the retry mechanism to and request made with 4o.
  4. Under action, select Change retry config and select Claude 3.7 as the fallback model. You can set a number of retries for each model - sometimes requests will randomly fail the first time even if the model provider is still available.

If your application makes requests with several different models and you want a different fallback for each, you will need to create a separate rule for each model and its corresponding fallback.

​
Step 2 - Creating a ruleset

We’ll now create a ruleset to apply to everyone and trigger our “fallback” rule.

  1. Head to the rulesets page (again accessible from the nav bar).
  2. Click to create a new ruleset, giving it a name and description.
  3. Under conditions, select the Is Group type. This applies to any request where a “group” is defined in the metadata. N.b. Waystone currently doesn’t have an “always” ruleset condition, but will soon.
  4. Select the rule we just created and add it to the ruleset.

​
Step 3 - Include user metadata in request

The Waystone API is a superset of the OpenAI API. We make use of the metadata field to identify users and apply the arbitrary metadata you define to them.

To assign user metadata, simply add the user’s unique ID in your application to the Waystone request and the “group” (e.g. a team or organisation. If you application does not have these entities, you can reuse the user ID). This metadata is sent as a JSON string in the metadata field of the request.

  const client = new OpenAI({
    apiKey: process.env['WAYSTONE_API_KEY'],
    baseURL: 'https://waystone.run/api/gateway/v1',
  });

  const userMetadata = {
    user: userId,
    group: groupId,
  }

  const response = await client.responses.create({
    model: "openai/gpt-4.1",
    input: "Your super special prompt",
    metadata: { waystone: JSON.stringify(userMetadata) }
  });

You don’t need any special code in your application to handle the fallback - Waystone manages the retry and model fallback for you.