X-Ray Vision for AI Agents: How We Built Production-Grade Observability into AACFlow

The first rule of our codebase: never use console.log. Every log statement goes through createLogger, our structured logging wrapper:

Every log entry is a structured object with:

• Timestamp — when it happened
• Level — info, warn, error, debug
• Context — execution ID, workflow ID, user ID, node ID
• Payload — the actual data relevant to the event

This structure means logs are queryable. Want to find every error from gmail_send in the last 24 hours? Query it. Want to trace a specific execution from start to finish? Filter by execution ID. This is impossible with unstructured console.log statements.

Execution Traces: The Full Story

When a workflow runs, every step is recorded in an execution trace. This is not a log file you grep through. It's a structured, visual timeline:

Every node shows:

• Input — what data went in
• Output — what data came out
• Timing — how long it took
• Cost — how many credits/tokens it consumed
• Status — success, failure, retry, skipped

When an agent does something unexpected, you don't guess. You trace.

Real-Time Streaming

Traces aren't just available after execution. They stream in real-time while the workflow runs. Through the same Socket.IO infrastructure that powers multiplayer collaboration, execution events are pushed to the UI as they happen.

A user watching a workflow run sees:

• Nodes light up green as they complete
• Timing bars fill up in real-time
• Error nodes glow red immediately
• The current active node pulses

This turns debugging from "run → wait → look at logs → guess → repeat" into "run → watch → see the exact point of failure → fix."

Error Recovery Visibility

When the executor triggers retry logic, the trace shows it clearly:

This transparency matters for debugging. If a workflow is slow, you can see: is it spending time on actual work, or on retries? If retries are common for a particular integration, you can adjust the rate limiting or switch providers.

Metrics Dashboard

Beyond individual traces, AACFlow provides aggregate metrics:

Success rate — what percentage of workflow executions complete successfully?

Latency distribution — p50, p95, p99 execution times over the last hour/day/week.

Error breakdown — which blocks fail most often? Which error types are most common?

Credit consumption — how many credits are being spent, on which providers, for which workflows?

Active executions — how many workflows are running right now? How many are paused waiting for human input?

These metrics power operational decisions: "We should add more retries for the Stripe integration — its error rate is 3x the average." "Our GPT-4o spending is 60% of the budget — can we route more traffic to Haiku?"

Audit Trail: Compliance and Security

For enterprise deployments, observability extends to an audit trail:

• Who triggered an execution and when
• Who modified a workflow and what changed
• Who accessed what data through which agent
• Who approved or rejected a human-in-the-loop decision

Audit logs are immutable and exportable. They're the foundation for SOC 2, ISO 27001, and GDPR compliance. Every action in the system leaves a permanent, queryable trace.

The createLogger Philosophy

Our logging infrastructure follows strict conventions:

No console.log — ever. The linter enforces this. Every log must go through createLogger.

Structured, not string interpolation. logger.info('User logged in', { userId }) — not logger.info(\User $ logged in`)`. Structured logs are queryable; strings are not.

Context propagation. Execution IDs, workflow IDs, and user IDs are automatically attached to every log in the execution chain. You never wonder "which execution was this log from?"

Level discipline. debug for development details. info for normal operations. warn for recoverable issues. error for failures that need attention. No mixing levels.

Performance awareness. Logging is async and batched. It never blocks the execution thread. In high-throughput scenarios, log levels can be dynamically adjusted without restarting the service.

What We Learned

Observability must be built in from day one. Retrofitting tracing onto a system that wasn't designed for it is a nightmare. Every component we build starts with the question: "How will we observe this in production?"

Logs without queryability are noise. The raw volume of logs from 10 million monthly executions is enormous. Without structured logging and indexed fields, finding the relevant information is like searching for a needle in a haystack — blindfolded.

Real-time traces reduce debugging time by 10x. The difference between "run → wait 2 minutes → read logs → guess" and "run → see failure in real-time → click the node → see exact error" is night and day. Our users debug workflows in seconds, not hours.

Metrics drive better decisions than intuition. When we looked at the metrics, we discovered that 70% of retries were for just 3 integrations. We invested engineering time in improving those integrations specifically — and reduced overall failure rate by 40%.

What's Next

• AI-powered trace analysis — an agent that reads your failed traces and suggests fixes
• Alerting — configurable thresholds that trigger notifications (Slack, email, Telegram) when error rates spike or latency degrades
• Distributed tracing — for multi-agent workflows where execution spans multiple services
• Custom dashboards — user-built metric views for specific workflows or teams

Observability isn't exciting in a demo. It's not flashy. But it's what separates production infrastructure from toys. And AACFlow is production infrastructure.

— Alexandr Chibilyaev