The Sync Engine: How AACFlow Keeps 170+ Data Sources Fresh Without Melting Your Server

That's the happy path. The devil, as always, is in the details.

Incremental Sync: Don't Re-Sync the Entire Universe

The most important optimization in the engine is incremental sync. Without it, every sync run would re-fetch, re-hash, and re-embed every document — even if 99.9% of them haven't changed.

Connectors that declare supportsIncrementalSync: true receive a lastSyncAt timestamp when listDocuments is called:

The connector filters at the API level — meaning only changed documents travel over the network. A knowledge base with 50,000 Confluence pages where 12 were edited today? Only 12 documents get fetched.

For connectors that don't support server-side filtering (and many don't, or implement it unreliably), the engine falls back to content hashing.

Content Hashing: The Universal Change Detector

Every document stored in the knowledge base carries a contentHash. The sync engine computes it on first sync and stores it alongside the document. On every subsequent sync:

1. Connector returns a document (whether from incremental filter or full listing)
2. Engine computes the hash of the incoming content
3. Engine compares against the stored hash
4. If incomingHash === storedHash → document is unchanged, skip processing
5. If incomingHash !== storedHash → document was modified, re-embed and update

The hash function is deliberately fast and collision-resistant. We use SHA-256 on the normalized text content — not the raw API response, which might contain timestamps or metadata that changes without the actual content changing.

This is how we achieve zero wasted work. A sync run that touches 100,000 documents where 200 changed processes exactly 200 embeddings — not 100,000.

Cursor Pagination: Feeding the Beast

Some sources have millions of documents. Fetching them all in one API call is impossible — and would violate every rate limit on the planet. Every connector's listDocuments uses cursor-based pagination:

The engine runs this loop automatically. The connector just returns { documents, nextCursor, hasMore }. The engine handles:

• Rate limit pauses — if the connector hits a 429, the engine waits and retries
• Progress tracking — the UI shows "Syncing page 12 of ~50..."
• Partial failure recovery — if page 23 fails with a transient error, the engine retries from page 23, not from page 1

The cursor is opaque to the engine. It's an implementation detail of each connector — could be a page token, an offset, a timestamp, whatever the source API uses.

syncContext: Cross-Page Caching Without Global State

Here's a problem that bit us early: expensive lookups repeated on every page.

Take the Confluence connector. To extract clean text from Confluence pages, the connector needs to know the schema of labels, spaces, and content types. Fetching that schema on every page of a 500-page sync is 499 redundant API calls.

The solution: syncContext. A mutable object the engine creates at the start of a sync run and passes to every listDocuments and getDocument call. Connectors can stash expensive lookups there:

syncContext lives for the duration of a single sync run and is discarded afterward. No memory leaks. No stale caches between syncs. No global mutable state. Just clean, scoped caching.

contentDeferred: The Lazy Content Pattern

Some connectors deal with content that's expensive to fetch in bulk. A CRM might return 10,000 deals in a list endpoint — but each deal's full description, comments, and attachments require a separate API call. Fetching all 10,000 full documents on every sync would be prohibitively slow.

The contentDeferred flag solves this:

When the engine sees contentDeferred: true:

1. It stores the document skeleton immediately (title, metadata, tags)
2. For new documents only (not previously stored), it calls getDocument(externalId) to fetch the full content
3. For existing documents, it compares a lightweight hash (often computed from lastModified + key metadata fields) — only calling getDocument if the lightweight hash differs
4. Once full content is fetched, it's embedded and the contentDeferred flag is cleared

This is a massive optimization. On an initial sync of 50,000 CRM deals, the engine fetches 50,000 full documents — necessary. But on the next incremental sync, only 200 deals changed — so only 200 getDocument calls. The other 49,800 are skipped based on hash comparison.

Retry Logic: When APIs Misbehave

APIs fail. Networks drop. Rate limits bite. The sync engine's retry logic treats failures as normal, expected events — not exceptions.

Transient errors (429 Too Many Requests, 503 Service Unavailable, network timeout):

• Exponential backoff: 1s → 2s → 4s → 8s → 16s (configurable max)
• Jitter: ±25% random variance to prevent thundering herd
• Per-document retry: if one document fails on page 17, only that document is retried, not the entire page
• Max retries: configurable (default 5), after which the document is marked failed

Permanent errors (400 Bad Request, 403 Forbidden, 404 Not Found):

• No retry — logged immediately with the full error context
• The document is marked failed with the error message preserved for debugging
• The sync continues with remaining documents

Page-level failures:

• If an entire page fails (e.g., API returns 500 for the page request), the engine retries the page from the same cursor
• If page-level retries are exhausted, the sync is paused with a clear error
• The user can resume from the last successful page — no lost progress

Adaptive Rate Limiting: Learning the Limits

Every API has rate limits, and they're almost always lower than documented. The engine uses adaptive rate limiting — it doesn't trust docs, it learns by doing.

1. Start conservative — 1 request per second for unknown APIs
2. Ramp up — increase throughput by 50% every 10 successful requests
3. Back off on 429 — when a rate limit is hit, drop to 25% of current rate
4. Stabilize — converge on the maximum sustainable throughput for that specific API

Different connectors stabilize at different rates. Gmail settles at ~20 req/s. Confluence at ~5 req/s. Some Russian government APIs stabilize at 0.5 req/s. The engine learns each one independently and persists the learned rate for future syncs (with a slow decay to re-test periodically).

Parallel Sync: Many Connectors, One Engine

Users don't sync one connector at a time. They connect 10, 20, 50 knowledge sources and expect them all to stay fresh.

The engine runs parallel syncs — up to a configurable concurrency limit per workspace. Each connector sync runs in its own async context with its own rate limiter, its own retry state, and its own syncContext:

Resource isolation is per-connector. A rate-limited government API crawling at 0.5 req/s doesn't slow down the Gmail sync running at 20 req/s. Each connector's rate limiter is independent.

SyncResult: The Report Card

Every sync run produces a SyncResult:

This isn't just a log line. It's the contract between the sync engine and the user. A sync that reports { added: 0, updated: 5, deleted: 0, unchanged: 49995, failed: 0 } on a 50,000-document knowledge base is a triumph of efficiency. Five embeddings computed, 49,995 skipped, zero failures.

When docsFailed > 0, the engine preserves the error details per-document so the user can investigate. A failed document isn't silently dropped — it's flagged in the UI with the specific error.

Deletion Detection: What's No Longer There

A document that exists in the knowledge base but is absent from the source API has been deleted. The engine detects this after a full listing completes:

1. After processing all pages, the engine queries all stored externalId values for the connector
2. It compares against the set of externalId values returned by the source
3. Documents in the DB but not in the source are marked as deleted (soft-delete — flagged, not removed, so the vector index stays consistent for existing embeddings)
4. Deleted documents are excluded from search results but preserved for audit trails

Incremental sync complicates this. If you last synced at 2:00 PM and only asked for documents modified after 2:00 PM, documents deleted at 2:30 PM won't appear. The engine handles this by periodically running a "full reconciliation" pass — a complete listing that detects deletions — on a configurable schedule (daily by default, hourly for frequently-changing sources).

Conflict Resolution: When Multiple Users Edit the Same KB

The knowledge base is a shared resource. What happens when User A triggers a manual sync while the scheduled sync is still running? The distributed Redis lock prevents duplicate concurrent syncs. Only one sync runs at a time per connector.

But what about metadata edits? User A might edit a document's tags while the sync engine is updating the document's content. The engine uses optimistic concurrency: it reads the document's updatedAt before writing and compares it. If the document was modified by a user between read and write, the engine merges: user-edited metadata is preserved, engine-updated content is stored.

What We Learned Building It

APIs lie about their rate limits. Every single one. The only reliable source of truth is empirical testing. Our adaptive rate limiter was born from the painful experience of trusting docs.

Incremental sync is a minefield of edge cases. What if the source clock is skewed? What if modifiedAfter returns documents modified exactly at lastSyncAt (duplicate) or one second after (missed)? What if the source deletes and re-creates a document with the same ID? We handle all of these through a combination of content hashing, grace periods, and periodic full reconciliation.

Progress visibility is critical for trust. A sync that runs for 45 minutes with no visible progress feels broken. We invested heavily in real-time progress reporting: page numbers, document counts, estimated time remaining. Users trust what they can see.

Parallelism is a force multiplier. A workspace with 20 connectors syncs in the time it takes the slowest connector, not the sum of all connectors. This alone makes the difference between a usable platform and one where "sync" means "go make coffee."

The sync engine is invisible infrastructure. When it works — and it works 99.9% of the time across 10M+ monthly agent executions — nobody thinks about it. And that's exactly the point.

— Alexandr Chibilyaev