Multiple Chat Models in Spring AI: OpenAI + Ollama Guide

Learn how to use multiple chat models in Spring AI with OpenAI and Ollama. Configure ChatClient beans, add fallback, routing and A/B testing.

1. Introduction

Most Spring AI tutorials connect your app to one model and stop there. That works well for a demo. However, real projects rarely stay that simple. In this guide, we will set up multiple chat models in Spring AI, step by step.

In the previous article, we explored ChatClient and ChatModel with OpenAI. Now we will take the next step. We will connect OpenAI and Ollama in the same Spring Boot application. Then we will pick the right model for each request.

By the end of this guide, you will know:

• why teams use more than one chat model
• what Spring AI does by default, and where it falls short
• the way Spring AI 2.0 recommends to build one ChatClient per model
• how to inject the right ChatClient bean
• simple patterns for routing, fallback and A/B testing

Let us get started.

2. Why Use Multiple Chat Models?

Let me start with a simple question. Why would anyone want two models when one already works?

The honest answer is that no single model is best at everything. Some models are smart but slow and costly. Others are fast and cheap but weaker at deep reasoning. So in production, teams often mix them.

Here are five practical scenarios where this helps.

2.1 Task-Based Model Selection

Not every question needs your most powerful model. For example, “What is your refund policy?” is a simple lookup. On the other hand, “Compare these three loan offers and explain the risks” needs real reasoning.

So you can route requests like this:

• Complex reasoning goes to a powerful model like GPT-4o.
• Basic queries go to a lightweight model like GPT-4o-mini or a local Llama model.

As a result, you save money and reduce response time for simple requests. Your heavy model only works when it really needs to.

2.2 Fallback Strategy

External APIs fail. Sometimes the provider has an outage. At other times, you hit a rate limit or your network has a bad day.

With a fallback strategy, your app calls the primary model first. If that call fails, it switches to a second model automatically. The user still gets an answer instead of an error page.

I have worked on banking integrations for many years. Over there, a backup channel for every external call is standard practice. The same idea works very well for AI calls too.

2.3 A/B Testing

Choosing a model based on blog posts and benchmarks is risky. It is better to test with your own data and your own users.

In A/B testing, you send a portion of traffic to model A and the rest to model B. Then you compare:

• accuracy of the answers
• latency, which means response time
• cost per request
• user feedback, like thumbs up or thumbs down

After a week or two, you have real numbers. Then the decision becomes easy.

2.4 User Preference

Some apps let the user choose the model. You have probably seen this in chat tools with a model dropdown.

For instance, one user might prefer a local model for privacy. Another user might want the best quality and not care about speed. Supporting multiple models makes this choice possible.

2.5 Specialized Models

Different models have different strengths. One model may be great at writing code. Another may be better at creative writing or summaries.

Therefore, you can combine them in one app. A coding request goes to the coding model. A marketing copy request goes to the creative model. In short, each task gets the best tool for the job.

3. Spring AI Default Behavior

Before writing any code, let us understand what Spring AI gives us out of the box.

3.1 What Auto-Configuration Gives You

When you add a model starter such as spring-ai-starter-model-openai, Spring Boot creates a few beans for you. The two important ones are:

• a ChatModel bean, for example OpenAiChatModel
• a ChatClient.Builder bean, which comes from ChatClientAutoConfiguration

You simply inject the builder, call build(), and start chatting. Here is the familiar single-model setup:

@RestController
public class ChatController {
 
    private final ChatClient chatClient;
 
    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }
 
    @GetMapping("/chat")
    public String chat(@RequestParam String message) {
        return chatClient.prompt()
                .user(message)
                .call()
                .content();
    }
}

This code is clean and simple. For a single model, it is all you need.

3.2 Why One Builder Is Not Enough

Now imagine you add the Ollama starter too. Your app now has two ChatModel beans: OpenAiChatModel and OllamaChatModel. However, Spring AI still offers only one auto-configured ChatClient.Builder.

Which model should that single builder use? Spring cannot guess. The builder needs exactly one ChatModel bean, and now there are two. So the ChatController above breaks. At startup, Spring complains that it expected a single ChatModel bean but found two.

So the default behavior is great for simple use cases. But it is not enough when multiple models are involved.

4. What You Need to Do

This guide uses Spring AI 2.0, the current release. At a high level, you need four steps:

1. Add a starter for each model provider.
2. Create one ChatClient per model with the ChatClientBuilderConfigurer that Spring AI provides.
3. Mark one of those clients as @Primary.
4. Inject the right client wherever you need it, based on your business rules.

Let us walk through each step with a working example. We will use OpenAI as the cloud model and Ollama as the local model.

4.1 Add the Dependencies

First, add both model starters in your pom.xml. I am assuming the Spring AI BOM is already in place, as shown in the previous article.

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
 
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

With both starters present, Spring Boot creates an OpenAiChatModel bean and an OllamaChatModel bean. Both are switched on by default.

One small trap here. Do not set spring.ai.model.chat to a single value like openai. That property keeps only the named chat model and switches the others off.

4.2 Configure Both Models

Next, add the settings for each provider in application.properties:

# OpenAI settings
spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.model=gpt-4o-mini
 
# Ollama settings
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=llama3.2

A few quick notes here:

• Keep the API key in an environment variable. Never commit it to Git.
• Start Ollama locally and pull the model first with ollama pull llama3.2.
• These model names are only defaults. You can override them per client, as we will see soon.

4.3 A Quick Note for Spring AI 1.x Users

Spring AI 1.x uses slightly different property names. There, the model name sits under options, like spring.ai.openai.chat.options.model.

The 1.x docs also suggest a different setup. You set spring.ai.chat.client.enabled=false and build clients with ChatClient.create(). That approach still works in 2.0, and section 5.3 covers it.

5. Creating Multiple ChatClient Beans

Now comes the main part. We will create one ChatClient bean per model inside a configuration class.

5.1 The Configuration Class

The Spring AI 2.0 docs recommend building each client through ChatClientBuilderConfigurer. This keeps observability and your ChatClientBuilderCustomizer beans working. In short, your custom clients behave just like the auto-configured one.

@Configuration
public class ChatClientConfig {
 
    private final ChatClientBuilderConfigurer configurer;
    private final ObservationRegistry observationRegistry;
    private final ChatClientObservationConvention chatClientConvention;
    private final AdvisorObservationConvention advisorConvention;
    private final ToolCallingAdvisor.Builder<?> toolCallingAdvisorBuilder;
 
    public ChatClientConfig(
            ChatClientBuilderConfigurer configurer,
            ObjectProvider<ObservationRegistry> registry,
            ObjectProvider<ChatClientObservationConvention> clientConv,
            ObjectProvider<AdvisorObservationConvention> advisorConv,
            ObjectProvider<ToolCallingAdvisor.Builder<?>> toolBuilder) {
        this.configurer = configurer;
        this.observationRegistry =
                registry.getIfUnique(() -> ObservationRegistry.NOOP);
        this.chatClientConvention = clientConv.getIfUnique();
        this.advisorConvention = advisorConv.getIfUnique();
        this.toolCallingAdvisorBuilder = toolBuilder.getIfAvailable();
    }
 
    // Builds a ChatClient.Builder the same way auto-configuration does
    private ChatClient.Builder newBuilder(ChatModel chatModel) {
        ChatClient.Builder builder = ChatClient.builder(chatModel,
                observationRegistry, chatClientConvention,
                advisorConvention, toolCallingAdvisorBuilder);
        return configurer.configure(builder);
    }
 
    @Bean
    @Primary
    public ChatClient openAiChatClient(OpenAiChatModel openAiChatModel) {
        return newBuilder(openAiChatModel).build();
    }
 
    @Bean
    public ChatClient ollamaChatClient(OllamaChatModel ollamaChatModel) {
        return newBuilder(ollamaChatModel).build();
    }
}

It looks longer than a one-liner, but the idea is simple:

• The constructor collects what Spring AI has already prepared: the configurer, the observation registry and the tool-calling setup.
• The newBuilder() method creates a builder in the same way as the auto-configuration.
• Each @Bean method passes a different ChatModel to newBuilder().

Spring injects the auto-configured OpenAiChatModel and OllamaChatModel for us. The bean names come from the method names, so we get openAiChatClient and ollamaChatClient.

5.2 Why We Mark One Client as @Primary

The Spring AI docs recommend marking one ChatClient bean as @Primary when several ChatModel beans exist. It also gives you a sensible default.

With @Primary in place, a constructor that asks for a plain ChatClient gets the OpenAI client. Classes that need Ollama still ask for it with @Qualifier(“ollamaChatClient”).

Use @Primary carefully, though. It hides which model a class really talks to. For a team project, I still prefer explicit qualifiers in the important services. New developers can then see the model choice at a glance.

5.3 The Quick Way: ChatClient.create() and ChatClient.builder()

You will often see a shorter version in older tutorials and in the Spring AI 1.x docs:

// application.properties: spring.ai.chat.client.enabled=false
@Configuration
public class SimpleChatClientConfig {
 
    @Bean
    public ChatClient openAiChatClient(OpenAiChatModel openAiChatModel) {
        return ChatClient.create(openAiChatModel);
    }
 
    @Bean
    public ChatClient ollamaChatClient(OllamaChatModel ollamaChatModel) {
        ChatClient.Builder chatClientBuilder =
                ChatClient.builder(ollamaChatModel);
        return chatClientBuilder.build();
    }
}

Both factory methods are valid:

• ChatClient.create(chatModel) is a shortcut. It gives you a client with default settings.
• ChatClient.builder(chatModel) gives you a builder. You can add a system prompt, options and advisors before calling build().

This code still compiles and runs in Spring AI 2.0. However, the 2.0 docs point out a catch. These methods skip the auto-configured builder, so observability and customizer beans are ignored. As a result, you lose metrics and tracing for these clients.

Also, the property spring.ai.chat.client.enabled=false turns off the whole ChatClient auto-configuration. That includes ChatClientBuilderConfigurer. So pick one style and do not mix them.

My suggestion is simple. The quick way is fine for a demo or a learning project. For production, use the configurer approach from section 5.1.

One more warning from my side. Check the parameter names carefully when you copy code like this. A common typo is naming the parameter chatModel but passing ollamaChatModel to the builder. That code will not even compile.

5.4 Same Provider, Different Models

Here is a trick that many beginners miss. You do not need two providers to have two models. You can create two clients from the same OpenAiChatModel with different default options.

Add these methods inside ChatClientConfig, so they can reuse newBuilder():

@Bean
public ChatClient smartChatClient(OpenAiChatModel openAiChatModel) {
    return newBuilder(openAiChatModel)
            .defaultOptions(OpenAiChatOptions.builder()
                    .model("gpt-4o")
                    .temperature(0.2))
            .build();
}
 
@Bean
public ChatClient fastChatClient(OpenAiChatModel openAiChatModel) {
    return newBuilder(openAiChatModel)
            .defaultOptions(OpenAiChatOptions.builder()
                    .model("gpt-4o-mini")
                    .temperature(0.7))
            .build();
}

Notice that we do not call build() on the options. In Spring AI 2.0, defaultOptions() takes the options builder itself. In 1.x, you pass the finished options object instead.

This setup is perfect for task-based selection. Both clients share the same API key and HTTP connection. Only the model name and a few options differ.

5.5 Give Each Client Its Own Personality

The builder also lets you set a default system prompt for each client. This is useful for specialized models.

@Bean
public ChatClient codeChatClient(OpenAiChatModel openAiChatModel) {
    return newBuilder(openAiChatModel)
            .defaultSystem("You are a senior Java developer. "
                    + "Answer with short, working code examples.")
            .build();
}
 
@Bean
public ChatClient creativeChatClient(OllamaChatModel ollamaChatModel) {
    return newBuilder(ollamaChatModel)
            .defaultSystem("You are a friendly content writer. "
                    + "Keep the tone light and simple.")
            .build();
}

Now each client behaves differently even before you write a single user prompt. Also, your controllers stay clean because the setup lives in one place.

6. Injecting the Right ChatClient

At this point, we have several beans of the same type. So Spring needs a little help to pick the right one.

6.1 Use @Qualifier

The safest approach is @Qualifier with the bean name:

@RestController
@RequestMapping("/api")
public class MultiModelController {
 
    private final ChatClient openAiChatClient;
    private final ChatClient ollamaChatClient;
 
    public MultiModelController(
            @Qualifier("openAiChatClient") ChatClient openAiChatClient,
            @Qualifier("ollamaChatClient") ChatClient ollamaChatClient) {
        this.openAiChatClient = openAiChatClient;
        this.ollamaChatClient = ollamaChatClient;
    }
 
    @GetMapping("/openai")
    public String askOpenAi(@RequestParam String message) {
        return openAiChatClient.prompt().user(message).call().content();
    }
 
    @GetMapping("/ollama")
    public String askOllama(@RequestParam String message) {
        return ollamaChatClient.prompt().user(message).call().content();
    }
}

Without the qualifier, Spring quietly injects the @Primary client. That may not be the model you wanted. Sometimes Spring can match by parameter name instead. However, relying on that is fragile. An explicit @Qualifier makes your intent clear to every reader.

6.2 Inject All Clients as a Map

Spring has one more handy feature. If you inject a Map of String to ChatClient, Spring fills it with every ChatClient bean. The key is the bean name.

@Service
public class ChatClientRegistry {
 
    private final Map<String, ChatClient> clients;
 
    public ChatClientRegistry(Map<String, ChatClient> clients) {
        this.clients = clients;
    }
 
    public ChatClient get(String name) {
        ChatClient client = clients.get(name);
        if (client == null) {
            throw new IllegalArgumentException("Unknown model: " + name);
        }
        return client;
    }
}

This approach is very useful when the model name arrives at runtime. We will use it for the user preference scenario shortly.

6.3 Test the Endpoints

Run the application and try these two calls:

curl "http://localhost:8080/api/openai?message=What+is+Spring+AI"
curl "http://localhost:8080/api/ollama?message=What+is+Spring+AI"

If both calls return an answer, your multi-model setup is working. The first answer comes from OpenAI’s cloud. The second comes from Ollama on your own machine.

7. Implementing the Real-World Scenarios

Having two clients is just the foundation. Now let us use them for the scenarios from section 2.

7.1 Task-Based Routing

@Service
public class ModelRouterService {
 
    private final ChatClient smartChatClient;
    private final ChatClient fastChatClient;
 
    public ModelRouterService(
            @Qualifier("smartChatClient") ChatClient smartChatClient,
            @Qualifier("fastChatClient") ChatClient fastChatClient) {
        this.smartChatClient = smartChatClient;
        this.fastChatClient = fastChatClient;
    }
 
    public String ask(String message) {
        ChatClient client = isComplex(message)
                ? smartChatClient
                : fastChatClient;
        return client.prompt().user(message).call().content();
    }
 
    private boolean isComplex(String message) {
        String text = message.toLowerCase();
        return message.length() > 300
                || text.contains("compare")
                || text.contains("analyse")
                || text.contains("step by step");
    }
}

The isComplex() check is deliberately simple here. In a real app, you could look at message length, keywords, or the user’s plan. You could even ask a cheap model to classify the request first.

7.2 Fallback When the Primary Model Fails

@Service
public class ResilientChatService {
 
    private static final Logger log =
            LoggerFactory.getLogger(ResilientChatService.class);
 
    private final ChatClient primary;
    private final ChatClient fallback;
 
    public ResilientChatService(
            @Qualifier("openAiChatClient") ChatClient primary,
            @Qualifier("ollamaChatClient") ChatClient fallback) {
        this.primary = primary;
        this.fallback = fallback;
    }
 
    public String ask(String message) {
        try {
            return primary.prompt().user(message).call().content();
        } catch (Exception ex) {
            log.warn("Primary model failed, using fallback: {}",
                    ex.getMessage());
            return fallback.prompt().user(message).call().content();
        }
    }
}

This is the simplest fallback you can write. Notice that we log every failure. If the fallback triggers too often, something is wrong with your primary setup.

Keep in mind that the model clients already retry some failed calls on their own. So the fallback kicks in only after those retries run out. For OpenAI in Spring AI 2.0, you can tune this with spring.ai.openai.max-retries.

For production, consider a proper circuit breaker like Resilience4j. It stops calling a failing model for some time. That way, users do not wait for a timeout on every single request.

7.3 Simple A/B Testing

@Service
public class AbTestingChatService {
 
    private final ChatClient variantA;
    private final ChatClient variantB;
 
    public AbTestingChatService(
            @Qualifier("openAiChatClient") ChatClient variantA,
            @Qualifier("ollamaChatClient") ChatClient variantB) {
        this.variantA = variantA;
        this.variantB = variantB;
    }
 
    public ChatResult ask(String userId, String message) {
        boolean useA = Math.abs(userId.hashCode() % 100) < 50;
        ChatClient client = useA ? variantA : variantB;
 
        long start = System.currentTimeMillis();
        String answer = client.prompt().user(message).call().content();
        long latency = System.currentTimeMillis() - start;
 
        String variant = useA ? "A-openai" : "B-ollama";
        return new ChatResult(variant, answer, latency);
    }
 
    public record ChatResult(String variant, String answer, long latencyMs) {}
}

Two small details make this code useful. First, we hash the user ID instead of picking randomly. That keeps the same user on the same variant every time.

Second, we return the variant name and latency with the answer. Without them, you cannot tell later which model gave which result. Save these values in a table and compare them after a week.

7.4 Letting the User Choose

Here we reuse the ChatClientRegistry from section 6.2.

@RestController
public class UserChoiceController {
 
    private final ChatClientRegistry registry;
 
    public UserChoiceController(ChatClientRegistry registry) {
        this.registry = registry;
    }
 
    @GetMapping("/chat/{model}")
    public String chat(@PathVariable String model,
                       @RequestParam String message) {
        return registry.get(model + "ChatClient")
                .prompt()
                .user(message)
                .call()
                .content();
    }
}

Now a call to /chat/openAi uses the OpenAI client. Similarly, /chat/ollama uses the local model. Adding a new model later only needs one new bean in the configuration class.

In a real product, do not expose raw bean names to the browser. Instead, keep a small list of allowed model names and map them to beans.

7.5 Using Specialized Models

This scenario reuses the codeChatClient and creativeChatClient from section 5.5. The routing looks similar to task-based selection.

public String assist(String taskType, String message) {
    ChatClient client = switch (taskType) {
        case "code" -> codeChatClient;
        case "blog", "marketing" -> creativeChatClient;
        default -> fastChatClient;
    };
    return client.prompt().user(message).call().content();
}

The difference is in the goal. Task-based routing mostly saves cost. Specialized routing mostly improves quality, since each model plays to its strengths.

8. Best Practices and Common Mistakes

Let me share a few lessons that will save you debugging time.

8.1 Picking the Right Pair of Models

Before writing any routing code, decide which models you actually need. A common and practical pair is one cloud model and one local model.

• A cloud model, such as GPT-4o, gives strong answers with no hardware on your side. You pay per token, and your data leaves your network.
• A local model, such as Llama 3.2 on Ollama, costs nothing per request. Your data also stays on your machine. On the other hand, quality is lower and you need decent hardware.

Another good pair is a big and a small model from the same provider. This keeps one API key and one billing account. It suits task-based routing very well.

My simple rule is this. Pick the pair that matches your biggest pain point. If cost hurts, add a cheap model. For uptime problems, add a fallback from a different provider. And when privacy matters, add a local model.

8.2 Measure Every Model

Adding models without measuring them is just guesswork. So track a few numbers for each ChatClient from day one.

• average and worst-case response time
• error rate and number of fallbacks
• tokens used per request, which drives your bill
• user ratings, if your UI collects them

The ChatResponse object from Spring AI includes token usage in its metadata. You can read it with call().chatResponse() instead of call().content(). Then push these numbers to Micrometer or simply log them in the beginning.

After a few weeks, the data will tell you clearly which model earns its place. Sometimes you will even find that the cheaper model is good enough for everything.

8.3 Best Practices

• Keep all ChatClient beans in one configuration class. It becomes your single source of truth.
• Give beans clear names like smartChatClient or codeChatClient. Avoid names like client1.
• Log which model answered each request. This helps a lot during production issues.
• Test your prompts on every model. The same prompt can behave quite differently on two models.
• Put timeouts on slow models, so one model cannot block all your threads.

8.4 Common Mistakes

• Injecting ChatClient.Builder after adding a second model. The auto-configured builder cannot choose between two ChatModel beans.
• Setting spring.ai.model.chat=openai while you also need Ollama. That property keeps only the named chat model.
• Mixing the two styles. With spring.ai.chat.client.enabled=false, the ChatClientBuilderConfigurer bean disappears too.
• Missing @Qualifier on injection points. Spring then cannot decide which bean to use.
• Hardcoding API keys in application.properties and pushing them to Git.
• Falling back to a much weaker model without telling anyone. Users notice the drop in quality quickly.
• Running A/B tests without saving the variant name. The test then gives you no useful data.

9. FAQ’s on multiple chat models in Spring AI

10. Conclusion

Working with multiple chat models in Spring AI is easier than it first looks. The auto-configured ChatClient.Builder is great for one model. But once you add a second model, you build the clients yourself.

Here is a quick recap of the steps:

• Add the starters for each provider you need.
• Build one ChatClient per model with ChatClientBuilderConfigurer.
• Mark one client as @Primary to act as the default.
• Inject clients with @Qualifier, or collect them all in a Map.
• Build routing, fallback, A/B testing and user choice on top.

Start small with two models and one simple router. Then add fallback and metrics as your app grows. This way, you get flexibility without making the code hard to follow.

11. Further Reading

• What Is Spring AI Framework: ChatClient & ChatModel with OpenAI
• Spring AI Reference: ChatClient API https://docs.spring.io/spring-ai/reference/api/chatclient.html
• Spring AI Reference: OpenAI Chat https://docs.spring.io/spring-ai/reference/api/chat/openai-chat.html
• Spring AI Reference: Ollama Chat https://docs.spring.io/spring-ai/reference/api/chat/ollama-chat.html
• Spring AI 1.0 Reference: ChatClient API, for 1.x users https://docs.spring.io/spring-ai/reference/1.0/api/chatclient.html
• Resilience4j CircuitBreaker Documentation https://resilience4j.readme.io/docs/circuitbreaker