How to Cut AI API Costs with a Smart Router

AI API costs climb when every request goes to the same model, repeated prompts miss the cache, and retries have no ceiling. My Node.js 26.5.0 run checked the router’s utility, premium, cache, fallback, rate-limit, and cost branches with assertions.

Route by workload before trimming prompts

A router needs an explainable decision, so send bounded tasks such as classification, extraction, and short summaries to a utility model, then reserve the premium model for requests whose quality checks justify its price.

Use product metadata, user-selected modes, or a tested classifier to set requiresPremium. If you need context on model capabilities before choosing that boundary, the ChatGPT guide gives you the broader model and API vocabulary.

Prompt compression should be an exception because another model call adds latency and spend, and it can remove details the premium model needed.

Build a provider-neutral smart router

The implementation injects provider functions, so routing logic does not depend on one software development kit (SDK). Rate limiting blocks excess work, cache hits avoid a provider call, and only a failed premium request falls back to the utility tier.

export class SmartRouter {
  constructor({ utility, premium, maxRequests = 60, windowMs = 60_000 }) {
    this.providers = { utility, premium };
    this.maxRequests = maxRequests;
    this.windowMs = windowMs;
    this.requests = new Map();
    this.cache = new Map();
  }

  checkLimit(userId, now = Date.now()) {
    const state = this.requests.get(userId) ?? { count: 0, startedAt: now };

    if (now - state.startedAt >= this.windowMs) {
      state.count = 0;
      state.startedAt = now;
    }

    state.count += 1;
    this.requests.set(userId, state);
    return state.count <= this.maxRequests;
  }

  async respond({ userId, prompt, requiresPremium = false }) {
    if (!this.checkLimit(userId)) {
      return { source: "blocked", reply: "Rate limit exceeded" };
    }

    const tier = requiresPremium ? "premium" : "utility";
    const cacheKey = `${tier}:${prompt.trim().toLowerCase()}`;
    const cached = this.cache.get(cacheKey);
    if (cached) return { ...cached, source: `${tier}-cache` };

    try {
      const reply = await this.providers[tier](prompt);
      const result = { source: tier, reply };
      this.cache.set(cacheKey, result);
      return result;
    } catch (error) {
      if (tier === "utility") throw error;
      const reply = await this.providers.utility(prompt);
      return { source: "utility-fallback", reply };
    }
  }
}

export function estimateCost({ requests, inputTokens, outputTokens, inputRate, outputRate }) {
  return requests * ((inputTokens * inputRate + outputTokens * outputRate) / 1_000_000);
}

A process-local Map keeps the example runnable without packages. Production deployments need a shared rate-limit and cache store because separate Node.js workers do not share memory.

Exercise every branch before deployment

The assertion file uses deterministic provider doubles to test routing without API keys or billable calls, then calculates one cost scenario from published token rates.

import assert from "node:assert/strict";
import { SmartRouter, estimateCost } from "./router.mjs";

const utility = async (prompt) => `utility:${prompt}`;
const premium = async (prompt) => {
  if (prompt.includes("fail")) throw new Error("premium unavailable");
  return `premium:${prompt}`;
};

const router = new SmartRouter({ utility, premium, maxRequests: 4 });
const simple = await router.respond({ userId: "u1", prompt: "Summarize this" });
const cached = await router.respond({ userId: "u1", prompt: "Summarize this" });
const complex = await router.respond({ userId: "u1", prompt: "Review this contract", requiresPremium: true });
const fallback = await router.respond({ userId: "u1", prompt: "fail safely", requiresPremium: true });
const blocked = await router.respond({ userId: "u1", prompt: "one more" });

assert.equal(simple.source, "utility");
assert.equal(cached.source, "utility-cache");
assert.equal(complex.source, "premium");
assert.equal(fallback.source, "utility-fallback");
assert.equal(blocked.source, "blocked");

const premiumOnly = estimateCost({ requests: 10_000, inputTokens: 1_000, outputTokens: 300, inputRate: 5, outputRate: 30 });
const routed = estimateCost({ requests: 5_000, inputTokens: 1_000, outputTokens: 300, inputRate: 5, outputRate: 30 })
  + estimateCost({ requests: 5_000, inputTokens: 1_000, outputTokens: 300, inputRate: 0.25, outputRate: 2 });
const savings = ((premiumOnly - routed) / premiumOnly) * 100;

console.log(`simple=${simple.source}`);
console.log(`cached=${cached.source}`);
console.log(`complex=${complex.source}`);
console.log(`fallback=${fallback.source}`);
console.log(`blocked=${blocked.source}`);
console.log(`premium-only=$${premiumOnly.toFixed(2)}`);
console.log(`50-50-router=$${routed.toFixed(2)}`);
console.log(`savings=${savings.toFixed(1)}%`);
console.log("all assertions passed");

The command completed with every assertion passing, and cache hits, premium failures, and rate-limit responses appear separately so a green result cannot hide an untested route.

Calculate savings from your traffic mix

A percentage claim needs token counts, model rates, and a routing split. The OpenAI API pricing page listed the following standard rates on July 18, 2026, per one million text tokens.

The executed example models 10,000 requests with 1,000 input tokens and 300 output tokens each, where GPT-5.5 costs $140.00 for the full workload and a 50/50 split between GPT-5.5 and GPT-5 mini costs $74.25.

Your result changes with output length, cache hits, tool charges, regional processing, and the percentage of requests that pass the utility-tier quality check. Measure those inputs from usage logs before setting a savings target.

Use the cheaper controls before adding another model call

OpenAI’s cost guidance recommends fewer requests, fewer input and output tokens, and a smaller model that maintains the accuracy your task needs. The Batch API handles asynchronous jobs. Flex processing exchanges slower responses plus occasional resource unavailability for lower cost.

• Cache stable prefixes. Keep repeated instructions at the start of a prompt so provider-side caching can match them.
• Batch offline work. Evaluations, enrichment jobs, and bulk classification rarely need an interactive response.
• Cap output. A routing decision saves little when an unconstrained response consumes the larger share of the bill.
• Reject excess traffic. Sending abusive requests to a utility model keeps spending money on abuse.

A second provider can reduce dependency on one API, but it adds another failure contract and another set of output differences. The DeepSeek API setup guide shows the integration surface if you want to evaluate a separate utility adapter.

Protect correctness across tiers

A cheaper response only counts as a saving when it passes the same acceptance check. Store route, model, token usage, latency, cache status, fallback reason, and validation result for each request.

Keep premium fallbacks narrow. Authentication errors, malformed requests, and policy rejections should fail closed because another model cannot repair the request safely.

Move state out of the Node.js process

Redis or another shared store can enforce the limit across replicas and attach a time to live (TTL) to cached entries, though sensitive prompts need hashed keys and a review against your data-retention rules.

Evaluate the router with replay traffic

Replay a scrubbed sample of production requests through both tiers, score them with task-specific checks, and compare cost only after the utility output clears the quality threshold. One bounded request class gives you a safer starting point than routing the whole application.

Log one week of route decisions, then compare projected cost with the provider invoice and failed-quality rate. Keep the router only when lower spend survives that check.