Question
You have a Microsoft Foundry project that serves a high-volume chat app.
Most requests are simple FAQs, but some require advanced reasoning.
You need to reduce costs and latency for common queries, without degrading the quality of the responses to complex questions.
What should you do?
- A. Route all the requests to a smaller model.
- B. Use a model cascade that routes the requests to different models. Correct
- C. Increase the value of the max_tokens parameter for all the requests.
- D. Route all the requests to the most capable model.
Explanation
Routing each request to a model that fits its complexity sends simple FAQs to smaller, cheaper, faster models and keeps larger or reasoning models for the requests that need them, which is the pattern Foundry's model router implements. Sending everything to a smaller model would degrade the complex answers, and sending everything to the most capable model would keep costs and latency high for the common queries. Increasing max_tokens only allows longer outputs and does nothing to lower cost or latency.
