One Key, Fifteen Brains: Inside the AACFlow AI API Layer

The Solution: A Unified API Layer

Our AI API layer provides a single interface that abstracts away provider differences:

Behind that one function call, the layer:

1. Resolves the provider — based on preferences, availability, cost, and capability
2. Transforms the request — maps our unified format to the provider's native format
3. Handles authentication — injects the correct API key or token
4. Routes the request — sends it to the provider's endpoint
5. Transforms the response — maps the provider's response back to our unified format
6. Handles errors — retries on transient failures, falls back to alternative providers
7. Tracks usage — logs token consumption for billing and analytics

The agent never knows or cares which provider actually processed the request.

Intelligent Routing

The routing logic is the brain of the layer. Given a chat request, the router decides:

Explicit preference. If the workflow specifies "use Claude Opus," the router obeys. Full user control when you need it.

Capability matching. If the request includes tool calls, the router filters to providers that support native tool calling. If the request is a simple text completion, any provider qualifies.

Cost optimization. For bulk, low-stakes tasks (summarize 1,000 support tickets), the router picks the cheapest capable model. For a single high-stakes task (draft a legal contract), it picks the most capable model.

Load balancing. If the primary provider is rate-limited or slow, the router shifts traffic to an equivalent alternative provider automatically.

Fallback chains. Configurable per workspace: "Try Claude Sonnet first. If unavailable or too slow, fall back to GPT-4o. If that fails, use Gemini Pro. If everything fails, queue for retry."

The Credit System: Pay for What You Use

Underneath the routing layer is a weighted credit system. Every provider call costs credits, and the cost is proportional to the actual compute:

Users buy credits once and spend them across all providers. One wallet. One bill. No managing 15 different API keys and invoices.

Provider Adapters: The Translation Layer

Each provider has an adapter — a thin translation layer that implements our unified interface:

Adding a new provider means writing a ~200-line adapter. The router, credit system, error handling, and observability all work automatically with the new provider once the adapter is registered.

Streaming: Real-Time Responses

Agents don't always need to wait for the full response. For chat-like interactions, streaming delivers tokens as they're generated:

Not all providers support streaming, and those that do implement it differently. The adapter pattern handles this: providers that support streaming implement buildStreamRequest and parseStreamChunk. Providers that don't have their responses simulated as a single-chunk stream. The agent code doesn't change.

BYOK: Bring Your Own Key

Some users — especially enterprises — want to use their own API keys. They have negotiated rates with OpenAI, or compliance requirements that prohibit third-party key usage.

Our BYOK model supports this. Users can:

• Bring their own keys — the platform uses the user's keys for API calls
• Use AACFlow credits — the platform's keys, managed credits
• Mix and match — some providers via BYOK, others via AACFlow credits

When BYOK is active, the routing layer injects the user's key instead of the platform key. Credit deductions are skipped for BYOK calls. But all other platform features — routing, fallback, observability, error handling — still work.

The business model works because we provide value beyond token resale: the visual editor, the DAG executor, the collaboration, the observability. We win by being the best place to run agents, not just by being a middleman for tokens.

Observability: See Every Call

Every AI API call through the layer is logged:

• Which provider and model was used
• How many tokens were consumed (input + output)
• How long the call took (latency)
• What the request and response were (for debugging)
• What routing decision was made and why
• How many credits were deducted

This data powers the execution traces in the workflow viewer. When an agent makes a wrong decision, you can trace back to the exact LLM call, see the prompt, the response, and determine: was it a bad model choice? A bad prompt? A routing mistake?

What We Learned

Provider APIs change constantly. New models, deprecated endpoints, pricing updates, rate limit adjustments. We run automated tests against every provider every hour to catch breaking changes before users do.

Latency varies wildly. The same model from the same provider can respond in 200ms or 20 seconds. Our routing layer tracks latency percentiles per provider and factors them into routing decisions.

Token counting is imprecise. Every provider counts tokens differently. Our credit system uses provider-reported token counts when available, with fallback to our own tokenizer estimates. The difference is small enough that it averages out over thousands of calls.

Fallback is a superpower. During the March 2025 OpenAI outage, AACFlow agents continued working because the router automatically shifted to Anthropic and Google. Users didn't notice. That's the level of reliability production AI infrastructure requires.

What's Next

• Model performance benchmarking — automated quality evaluation so the router can pick not just the cheapest model, but the best-performing one for each task type
• Fine-tuned model hosting — support for user-provided fine-tuned models alongside the standard providers
• Multi-model ensembles — send the same request to multiple providers and select the best response
• Cost forecasting — predict monthly spend based on usage patterns before the bill arrives

The AI API layer is the nervous system of every AACFlow agent. It's invisible when it works — and that's the point.

— Alexandr Chibilyaev