Skip to content
Surfc Wiki

AI Ingestion

AI Ingestion & Tagging

CHANGE SUMMARY

  • Updated: Documented the managed Anthropic proxy path, quota enforcement, and the ai_usage_daily writer (src/api.js, src/supabase.js, supabase/functions/anthropic-proxy/index.ts, supabase/migrations/0004_ai_usage_tracking.sql, 0005_upsert_ai_usage_fn.sql).
  • Updated: Differentiated BYOK vs. managed payload flows and risks, and noted the UI affordances for managed-rate-limit surfacing.

Skill in use: ai-ingestion — documenting implemented AI flows, trigger points, payloads, persistence, and risks.

1. AI capabilities implemented

  • Image transcription (BYOK + managed): callTranscribeImage in src/api.js routes requests in two ways. If a user saved an Anthropic key, it posts base64 images directly to /v1/messages with model claude-haiku-4-5-20251001, retrying with an ASCII-only prompt on parse failures. Without a key, it sends { action: 'transcribe', payload, mimeType } plus the Supabase session to invokeAnthropicProxy, which calls the Edge Function (src/api.js, src/supabase.js).
  • Idea discovery (BYOK + managed): callDiscoverIdeas likewise either POSTs directly to Anthropic (BYOK) or forwards { action: 'discover', payload: noteText, customIdeas } to the proxy. In both cases canonical + custom idea lists are included, and client-side canonicalisation keeps behavior consistent regardless of path (src/api.js, src/hooks/useNoteActions.js). Tests in src/test/api.test.js still cover BYOK parsing.
  • Supabase Edge Function: supabase/functions/anthropic-proxy/index.ts now implements the managed path: it verifies the caller’s JWT, sums month-to-date usage via ai_usage_daily, enforces the free-tier limit (30 managed calls/user/month), proxies to Anthropic with the server key, and records successful usage via the upsert_ai_usage RPC.
  • Ingest adapters: src/ingest/manualAdapter.js and src/ingest/photoAdapter.js normalize manual text and camera captures respectively. photoAdapter compresses images (src/utils.js), calls callTranscribeImage, and annotates sourceMeta.
  • Rediscovery: useNoteActions.rediscoverIdeas can re-run callDiscoverIdeas on saved notes to refresh canonical tags while preserving custom ones (src/hooks/useNoteActions.js).

2. Trigger points in UI/app flow

  • Capture flow: <NoteForm /> invokes useNoteForm.handleImageSelected, which opens the camera/file input, compresses the file, and calls ingest(photoAdapter, image, { apiKey, session }). photoAdapter.acquire passes the session so the adapter can call the managed proxy whenever apiKey is empty (src/hooks/useNoteForm.js, src/ingest/photoAdapter.js).
  • Retry: retranscribeImage reuses capturedImage if the first call failed, still via callTranscribeImage.
  • AI tagging: Clicking “Discover Ideas” (NoteForm action row) triggers useNoteForm.discoverIdeas, which passes apiKey, customIdeas, and session to callDiscoverIdeas. Users without keys automatically run through the managed proxy; BYOK users continue to hit Anthropic directly (src/hooks/useNoteForm.js).
  • Rediscover: Long-pressing a note and choosing “Rediscover Ideas” runs useNoteActions.rediscoverIdeas; managed users now reach the proxy because the hook hands the Supabase session to callDiscoverIdeas (src/hooks/useNoteActions.js).
  • Rate-limit UX: Both useNoteForm and useNoteActions check err.isRateLimit (set by invokeAnthropicProxy) to display “Monthly limit reached — add your own key” guidance when the Edge Function returns HTTP 429 (src/hooks/useNoteForm.js, src/hooks/useNoteActions.js, src/supabase.js).

3. Data sent to external APIs

  • Managed path: invokeAnthropicProxy POSTs to Supabase Functions with the user’s JWT. The Edge Function then calls Anthropic server-side with Deno.env secrets, so note text/image data never leaves the Supabase boundary unless the managed function succeeds (src/supabase.js, supabase/functions/anthropic-proxy/index.ts).
  • Transcription payload (BYOK): photoAdapter sends image.mimeType and base64 bytes straight to Anthropic along with the two system prompts (primary + ASCII fallback). No note metadata accompanies the call beyond what Anthropic extracts visually (src/ingest/photoAdapter.js, src/api.js).
  • Payload size impact: compressImage emits JPEGs up to 1024 px on the long edge at 0.78 quality, keeping payloads well under 500 KB for typical book-page captures (src/utils.js:20-33, src/ingest/photoAdapter.js:16-33).
  • Idea tagging payload: Both managed and BYOK flows include the user’s note text plus the 102 canonical Great Ideas and any custom idea names/descriptions derived from IndexedDB. In managed mode those prompts travel browser -> Supabase Edge -> Anthropic; BYOK mode still transmits them directly to Anthropic with the anthropic-dangerous-direct-browser-access header (src/api.js, src/constants.js).
  • Headers & secrets: BYOK calls send the stored key verbatim from Dexie meta.apiKey. Managed calls never expose Anthropic keys to the browser; the Edge Function uses ANTHROPIC_API_KEY + SUPABASE_SERVICE_ROLE_KEY environment variables (supabase/functions/anthropic-proxy/index.ts).

4. Persistence of results

  • Transcription output: useNoteForm.handleImageSelected merges Anthropic text into noteText, sets notePage, records detectedAttribution, and stores capturedImage for preview; errors surface as transcribeError.
  • Idea tags: useNoteForm.discoverIdeas writes the returned array to pendingTags, which later becomes the note’s tags when saveNoteForm executes.
  • Saved note: saveNoteForm persists { text, tags, source, sourceMeta, imageDataUrl } via saveNote and enqueues cloud sync; uploaded images update imagePath when Supabase Storage returns a path.
  • Rediscover: useNoteActions.rediscoverIdeas overwrites canonical tags in Dexie and pushes an upsert via cloudWrite.
  • Usage ledger: Successful managed calls increment ai_usage_daily via recordUsage -> rpc('upsert_ai_usage', …) before responses return to the client. BYOK calls have no corresponding ledger entry, by design (supabase/functions/anthropic-proxy/index.ts, supabase/migrations/0004_ai_usage_tracking.sql, 0005_upsert_ai_usage_fn.sql).
  • State sharing: App.jsx threads AI state (loading/errors) to ReviewScreen and NoteForm for UX feedback.

5. Provenance model

  • Source fields: Dexie schema v5+ adds notes.source, sourceId, and sourceMeta; photoAdapter sets source: 'image' and sourceMeta containing case, title, author, page, while manual notes default to source: 'manual' (src/db.js, src/ingest/photoAdapter.js).
  • Deduping: ingest.filterDuplicates checks sourceId to avoid re-ingesting the same external highlight; current adapters return null, so dedupe is effectively disabled for camera/manual captures.
  • Attribution UI: useNoteForm exposes detectedAttribution so the NoteForm component can suggest adding a book with the detected title/author, linking AI output back to the library.
  • Supabase parity: supabase/migrations/0001_initial_schema.sql, 0002_notes_add_source.sql, and 0003_notes_add_source_ingest.sql store source, source_id, and source_meta, so provenance data syncs across devices once saved. Managed usage counters live separately in ai_usage_daily and are never synced to devices (supabase/migrations/0004_ai_usage_tracking.sql).

6. Risks and unknowns

  • API key exposure persists for BYOK: Keys remain plaintext in IndexedDB (meta.apiKey) and flow directly to Anthropic for BYOK users; only managed users benefit from server-side proxying (src/db.js, src/api.js).
  • Quota visibility: Managed usage is enforced server-side but never surfaced in the UI beyond a generic toast/message. Users cannot see remaining calls or token counts even though ai_usage_daily stores them (src/supabase.js, supabase/functions/anthropic-proxy/index.ts).
  • Error handling: Failures set transcribeError/aiError but do not queue retries; once a note saves without tags, there is no background job to retry automatically.
  • Future adapters: Comments mention Readwise/Kindle ingestion, but no implementations exist; dedupe logic must be extended before enabling them.
  • Unknown throughput: There are no tests covering callTranscribeImage; only callDiscoverIdeas parsing is covered, leaving OCR prompts unvalidated.
  • Image limits: compressImage resizes to max 1024 px but there is no guardrail for extremely large originals or different mime types.
  • Latency: callTranscribeImage targets claude-haiku-4-5-20251001 with max_tokens: 500. JSON parse failures still trigger a second Anthropic round-trip with an ASCII-only system note (src/api.js). The UI wait inside useNoteForm.handleImageSelected remains blocking with no cancel path (src/hooks/useNoteForm.js).
  • Managed session dependency: invokeAnthropicProxy now calls supabase.auth.getSession() immediately before hitting the Edge Function and only then sets Authorization: Bearer ${session.access_token}. This per-call refresh removed the stale-JWT race we saw earlier; the remaining failure mode is when getSession() returns null (user logged out or refresh token invalid), which surfaces as “Session expired — please sign in again” (src/supabase.js:150-196).
  • Bias handling: The system prompt enforces ASCII quotes and instructs to ignore unmarked text, but there is no auditing of Anthropic output fidelity.