祝融说。

FORM NOT VOID, MIND NO CORE

Offline Sync Engine

2026.01.20

An offline sync engine for Local-First architecture, optimized for low-bandwidth environments.

Chinese version | English

A complete offline sync solution designed with a Local-First architecture. Applications can run entirely offline, using local storage as the primary data source, while automatically synchronizing with the server in the background. Optimized for unstable network conditions—such as 2G/3G networks in Africa—with support for data compression, resumable transfers, and intelligent conflict resolution.

Build Status TypeScript License

✨ Features

Core Capabilities

  • 🌐 Full Offline Support - Built on IndexedDB local storage, fully functional offline
  • 🔄 Automatic Sync - Automatically synchronizes when network is detected
  • ⚡ Incremental Sync - Transmits only changed data, conserving bandwidth
  • 🗜️ Outbox Pattern - Intercepts write operations, queues locally, ensures reliable sync
  • 🧠 Intelligent Conflict Resolution - Last-Write-Wins (LWW) + Vector Clocks
  • 📱 Cross-Platform - Built on RxDB, supports Web and mobile platforms

Advanced Features

  • 📦 Data Compression - MessagePack + DEFLATE, reduces data size by 40–60%
  • 📤 Resumable Transfers - Full TUS protocol implementation, supports large file uploads
  • ⚡ Performance Optimization - Batch operations, indexed optimizations, query caching
  • 🔌 Real-Time Push - WebSocket server-side notification push
  • 🛡️ Type Safety - End-to-end TypeScript support

📐 Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                        Client Application                     │
│  ┌────────────────────────────────────────────────────────┐ │
│  │                    UI Layer (React)                     │ │
│  └──────────────────────┬─────────────────────────────────┘ │
│                         │                                   │
│  ┌──────────────────────▼─────────────────────────────────┐ │
│  │             Offline SDK (@offline-sync/sdk)             │ │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌────────┐     │ │
│  │  │ Storage  │  │ Network  │  │ Outbox  │  │ Sync   │     │ │
│  │  │ Layer    │  │ Manager  │  │ (Queue) │  │ Manager│     │ │
│  │  │ (RxDB)  │  │          │  │         │  │        │     │ │
│  │  └────┬────┘  └────┬────┘  └────┬────┘  └───┬────┘     │ │
│  │       │            │            │            │         │ │
│  │  ┌───▼────────────▼────────────▼────────────▼───────┐  │ │
│  │  │           IndexedDB (Browser Local Storage)        │  │ │
│  │  └──────────────────────────────────────────────────┘  │ │
│  └────────────────────────────────────────────────────────┘ │
└──────────────────────────────┬──────────────────────────────┘
                               │ HTTPS (compressed transfer)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                    Sync Gateway Server                      │
│  ┌──────────────┐  ┌──────────┐  ┌─────────┐  ┌────────┐    │
│  │   Gateway    │  │ Applicator│  │ Arbiter │  │   TUS  │    │
│  │  (Routing)   │  │(Apply Ops)│  │(Conflict│  │(Resume)│    │
│  │              │  │           │  │ Resolution)│        │    │
│  └──────┬───────┘  └────┬─────┘  └────┬─────┘  └────┬───┘   │
│         │               │             │             │       │
│  ┌──────▼──────────────▼─────────────▼─────────────▼───┐    │
│  │                 CouchDB (Primary Database)           │   │
│  │  - todos, products, customers, orders                │   │
│  │  - _changes feed for incremental sync                │   │
│  │  - Mango Query support                               │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/iannil/offline-sync-engine.git
cd offline-sync-engine

# Install dependencies
pnpm install

Run the Development Server

# Start the server (port 3000)
pnpm dev:server

# Start the client demo (port 5173)
pnpm dev:client

Build

# Build the SDK
pnpm --filter @offline-sync/sdk build

# Build the server
pnpm --filter @offline-sync/server build

# Build the demo app
pnpm --filter @offline-sync/client-demo build

💻 Usage Examples

Basic SDK Usage

import { OfflineClient } from '@offline-sync/sdk';

// Initialize the client
const client = new OfflineClient({
  database: { name: 'my-app' },
  sync: {
    enabled: true,
    url: 'https://api.example.com/sync',
    interval: 30000,  // Sync every 30 seconds
    enableCompression: true,
  },
});

// Wait for the client to be ready
await client.initialize();

// Get the database
const db = client.getDatabase();

// Create a to-do (offline + automatic sync)
const todo = await db.todos.insert({
  id: 'todo-1',
  text: 'Learn the offline sync engine',
  completed: false,
  createdAt: new Date().toISOString(),
  updatedAt: new Date().toISOString(),
});

// Trigger sync manually
await client.getSyncManager().triggerSync();

// Listen to sync state changes
client.getSyncManager().onStateChange((state) => {
  console.log('Syncing:', state.isSyncing);
  console.log('Pending items:', state.pendingCount);
});

TUS Resumable Uploads

import { createTusUpload } from '@offline-sync/sdk/storage';

// Create a file upload
const uploader = createTusUpload({
  endpoint: 'https://api.example.com/api/tus',
  data: file,
  metadata: {
    filename: file.name,
    type: file.type,
  },
  chunkSize: 5 * 1024 * 1024,  // 5MB chunks
  onProgress: (sent, total) => {
    console.log(`Progress: ${(sent / total * 100).toFixed(1)}%`);
  },
});

// Start upload
const uploadUrl = await uploader.start();

// Pause upload
uploader.pause();

// Resume upload (supports resumable transfer)
await uploader.resume();

Server API

# Push local operations to server
curl -X POST https://api.example.com/api/sync/push \
  -H "Content-Type: application/msgpack+deflate" \
  -H "Accept: application/msgpack+deflate" \
  --data-binary '@payload.bin'

# Pull server changes
curl "https://api.example.com/api/sync/pull?since=1234567890" \
  -H "Accept: application/msgpack+deflate"

# TUS create upload
curl -X POST https://api.example.com/api/tus \
  -H "Tus-Resumable: 1.0.0" \
  -H "Upload-Length: 1024000" \
  -H "Upload-Metadata: filename dGVzdC5qcGc="

📦 Package Structure

offline-sync-engine/
├── packages/
│   ├── sdk/              # Client SDK
│   │   ├── src/
│   │   │   ├── storage/     # Storage module
│   │   │   ├── network/     # Network manager
│   │   │   ├── outbox/      # Offline queue
│   │   │   ├── sync/        # Sync manager
│   │   │   └── client/      # Client entrypoint
│   │   └── package.json
│   │
│   ├── server/           # Sync gateway server
│   │   ├── src/
│   │   │   ├── gateway/     # Sync gateway
│   │   │   ├── applier/     # Operation applier
│   │   │   ├── arbiter/     # Conflict arbiter
│   │   │   ├── database/    # Database layer
│   │   │   └── tus/         # TUS protocol
│   │   └── package.json
│   │
│   └── client-demo/       # Demo application
│       ├── src/
│       │   ├── components/
│       │   └── db/
│       └── package.json
│
├── docs/                 # Documentation
├── pnpm-workspace.yaml  # Monorepo configuration
└── package.json

🔧 Configuration

SDK Configuration

interface OfflineClientConfig {
  // Database configuration
  database: {
    name: string;              // Database name
  };

  // Sync configuration
  sync?: {
    enabled: boolean;         // Enable sync
    url: string;              // Sync server URL
    interval?: number;        // Sync interval (milliseconds)
    batchSize?: number;       // Batch size
    enableCompression?: boolean;  // Enable compression
    enableWebSocket?: boolean;    // Enable WebSocket
  };

  // Outbox configuration
  outbox?: {
    maxRetries?: number;      // Maximum retry attempts
    initialDelay?: number;    // Initial retry delay (milliseconds)
    maxDelay?: number;        // Maximum retry delay (milliseconds)
  };
}

Server Configuration

# Environment variables
COUCHDB_URL=http://localhost:5984
COUCHDB_USERNAME=admin
COUCHDB_PASSWORD=password
COUCHDB_DB_PREFIX=offline-sync
PORT=3000
HOST=0.0.0.0

📚 API Documentation

SDK Exports

// Client
import { OfflineClient } from '@offline-sync/sdk/client';

// Storage
import {
  createDatabase,
  getDatabase,
  todoSchema,
  productSchema,
} from '@offline-sync/sdk/storage';

// Query
import {
  findAll,
  findById,
  findWhere,
  paginate,
  count,
  QueryBuilder,
} from '@offline-sync/sdk/storage';

// Compression
import {
  CompressionService,
  compress,
  decompress,
} from '@offline-sync/sdk/storage';

// TUS Protocol
import {
  createTusUpload,
  uploadFile,
  TusUploader,
} from '@offline-sync/sdk/storage';

// Testing
import {
  benchmarkWrite,
  benchmarkRead,
  benchmarkQuery,
  testCapacity,
} from '@offline-sync/sdk/testing';

// Types
import type { Todo, Product, OutboxAction, NetworkStatus } from '@offline-sync/sdk';

Server Endpoints

EndpointMethodDescription
/healthGETHealth check
/api/sync/pushPOSTPush local operations
/api/sync/pullGETPull remote changes
/api/sync/:collectionGETGet collection data
/api/sync/:collection/:idGETGet single document
/api/applier/applyPOSTApply single operation
/api/applier/batchPOSTBatch apply operations
/api/arbiter/checkPOSTConflict detection
/api/arbiter/resolvePOSTLWW conflict resolution
/api/arbiter/resolve/mergePOSTField-level merge
/api/tusPOSTCreate upload
/api/tus/:idPATCHUpload chunk
/api/streamWSReal-time push

🧪 Development

Environment Requirements

  • Node.js >= 18
  • pnpm >= 8
  • CouchDB >= 3.0 (optional, for production)

Development Commands

# Install dependencies
pnpm install

# Start development servers
pnpm dev:server  # server
pnpm dev:client  # client

# Run tests
pnpm test

# Lint and format code
pnpm lint
pnpm format

Local CouchDB Development

# Start CouchDB with Docker
docker run -d \
  --name couchdb \
  -p 5984:5984 \
  -e COUCHDB_USER=admin \
  -e COUCHDB_PASSWORD=password \
  couchdb:3

Code Standards

  • Write code in TypeScript
  • Follow ESLint rules
  • Add unit tests for new features
  • Update relevant documentation

🔗 Technology Stack

CategoryTechnology
Frontend FrameworkReact + TypeScript
Local DatabaseRxDB + Dexie (IndexedDB)
Backend FrameworkFastify (Node.js)
Primary DatabaseCouchDB
Data SerializationMessagePack
Data CompressionDEFLATE (pako)
Resumable TransfersTUS Protocol v1.0.0
Real-Time CommunicationWebSocket
Package Managerpnpm workspaces
Build Toolstsup (library) + Vite (application)
Testing FrameworkVitest

📄 Acknowledgments

This project is built upon the following excellent open-source projects:

writings