Skip to content
Surfc Wiki

Local Data + Sync

Local Data + Sync

CHANGE SUMMARY

  • Updated: Added the runtime schema probe that syncFromCloud runs before any flush/merge work, referencing src/hooks/useAuth.js and src/supabase.js.
  • Updated: Clarified that SCHEMA_VERSION now matches Dexie v6 and is enforced by src/test/export-import.test.js, removing the stale warning about version lag.

Skill in use: local-data-sync — capturing Dexie schema, migrations, write/sync paths, and risks per AGENTS.md.

1. Local data model

  • Dexie database: src/db.js instantiates SurfcDB with tables meta, books, notes, customIdeas, and outbox.
  • Entities:
    • books: { id, title, author, createdAt, updatedAt, deleted } with indexes on id and timestamps; deleting a book cascades soft-delete to related notes.
    • notes: { id, bookId, text, page, tags[], imagePath, imageDataUrl, source, sourceId, sourceMeta, createdAt, updatedAt, deleted } plus a multiEntry index on tags for idea queries.
    • customIdeas: { id, name, description, createdAt, updatedAt, deleted }.
    • meta: key/value store for apiKey and lastSyncAt.
    • outbox: queue of offline mutations { id (auto), table, recordId, payload, createdAt }.
classDiagram
  class Book {
    id: text (PK)
    title: text
    author: text
    createdAt: ms epoch
    updatedAt: ms epoch
    deleted: 0/1
  }
  class Note {
    id: text (PK)
    bookId: text | null
    text: string
    page: string
    tags: string[]
    imagePath: text | null
    imageDataUrl: base64 | null
    source: manual|image|future
    sourceId: string | null
    sourceMeta: object
    createdAt: ms epoch
    updatedAt: ms epoch
    deleted: 0/1
  }
  class CustomIdea {
    id: text (PK)
    name: text
    description: text
    updatedAt: ms epoch
    deleted: 0/1
  }
  class Outbox {
    id: auto
    table: text
    recordId: text | null
    payload: json
    createdAt: ms epoch
  }
  Note --> Book : optional bookId

2. Schema evolution

  • v1: base tables without updatedAt, deleted, outbox, or provenance fields.
  • v2: adds updatedAt, deleted, imagePath, and introduces outbox; upgrade migrates existing rows.
  • v3: introduces notes.source.
  • v4: adds outbox.recordId for dedupe.
  • v5: adds notes.sourceId + sourceMeta.
  • v6: adds Dexie *tags multiEntry index.
  • All versions are defined inline in src/db.js; Dexie upgrades mutate existing rows to seed new fields.
  • SCHEMA_VERSION now equals 6, matching Dexie v6, and Vitest enforces the guard in src/test/export-import.test.js, so exports include provenance/tag fields without manual fixes.

3. Write path (local → cloud)

  • Books/custom ideas: saveBook / saveCustomIdea stamp updatedAt and deleted:0; deleteBook runs a Dexie transaction to tombstone the book and related notes (src/db.js).
  • Notes: saveNote persists note text/tags/provenance locally, defaulting source: 'manual' unless capture metadata overrides it; setNoteImagePath later stores the Supabase storage path.
  • Meta: saveApiKey and saveLastSync keep user-specific metadata in Dexie meta.
  • Hook integration: useNoteForm.saveNoteForm creates Dexie notes, uploads images via uploadImage, and immediately calls cloudWrite('notes', saved) so Supabase reflects the same payload (src/hooks/useNoteForm.js).
  • Outbox: useAuth.cloudWrite enqueues payloads via enqueue when offline or Supabase rejects a write; entries include recordId to collapse retries (src/hooks/useAuth.js, src/db.js).

4. Sync path

  • Schema guard (session start): syncFromCloud checks schemaProbed.current before any writes and calls probeCloudSchema; mismatches set syncStatus and abort the sync so the user can run npm run check:schema (src/hooks/useAuth.js, src/supabase.js, package.json).
  • Outbound (local → Supabase): useAuth.syncFromCloud retrieves outbox entries, calls flushOutbox (src/supabase.js), which groups mutations via collapseOutboxItems, attempts Supabase upserts, and clears successful IDs. Tests in src/test/outbox.test.js cover delete stickiness and ordering.
  • Inbound (Supabase → local): After flushing, syncFromCloud downloads every table via fetchAllCloud (currently full-table) and feeds the payload to mergeCloudRecords. That function compares updated_at vs. local.updatedAt, overwriting local rows when the cloud version wins and skipping deletes for rows never stored locally (src/db.js).
  • Images: Post-merge, syncFromCloud iterates local notes; if imagePath exists but imageDataUrl is missing, it downloads from Supabase Storage and patches the row via setNoteImagePath + db.notes.update.
  • State refresh: loadAll reloads filtered, soft-delete-free collections and updates React state; saveLastSync records the timestamp for display.
  • Online/offline: useAuth listens to window events to trigger syncFromCloud upon reconnection, ensuring outbox flush before merging.

5. Conflict resolution

  • Last-write-wins: mergeCloudRecords compares updated_at to local.updatedAt; newer wins per entity. Soft-deleted cloud records (deleted truthy) overwrite local rows, but deletion is skipped if the record never existed locally (prevents ghost resurrections).
  • Outbox collapse: collapseOutboxItems sorts by createdAt, groups by table + recordId, merges payload fields, and treats deleted:1 as sticky so tombstones override earlier edits; tests cover ordering edge cases (src/test/outbox.test.js).
  • Source dedupe: Comments note future dedupe on sourceId, but currently notes_source_id_idx is non-unique on Supabase, and Dexie stores duplicates; dedupe must occur at the app level.
  • Rediscovery: useNoteActions.rediscoverIdeas preserves custom tags while replacing canonical ones, ensuring deterministic merges when AI re-tags.

6. Export/import

  • buildExport packages { books, notes, customIdeas } alongside _syntopicon flag, schemaVersion, and ISO timestamp (src/db.js).
  • parseImport validates the flag and returns arrays, throwing if the file is not a Surfc export.
  • importMerge inserts imported rows only if they do not already exist; importReplace clears each table then bulk inserts. Both run inside Dexie transactions and stamp updatedAt defaults if missing.
  • useSettings exposes export/import through the modal UI, wiring buttons to downloadJSON, file input parsing, and confirmMerge / confirmReplace flows (src/hooks/useSettings.js).
  • There is no per-field conflict UI; merges rely solely on ID uniqueness.

7. Risks and unknowns

  • One-shot schema probe: schemaProbed.current flips to true before the probe completes and never resets after probeCloudSchema fails, so users must reload to rerun the guard (src/hooks/useAuth.js).
  • Single outbox queue: Books, notes, and custom ideas share one queue without per-table pacing; a flood of note edits could block other entity types until flush completes.
  • Full-table fetch: fetchAllCloud pulls entire tables every sync; there is no incremental updated_at checkpoint despite fetchSince existing, so large datasets could slow hydration.
  • No conflict UI: Last-write-wins may hide collisions between devices; users cannot inspect or resolve divergent edits beyond timestamps.
  • Unknown multi-tab behavior: Dexie singleton is shared, but there is no visibility into concurrent tabs updating the same records (risking stale UI updates).
  • Image consistency: imageDataUrl is preserved during merges, but uploads happen opportunistically; missing imagePath plus no upload retry means photos can remain stranded locally without warning.