app/docs/fixes/galaxy-graph-fix.md

Galaxy Graph Visualization Fix

Problems

1. Invalid URL Error: ThoughtGalaxy component was failing with:

TypeError: Failed to construct 'URL': Invalid URL
    at parseUrl (surreal.ts:745:14)
    at Surreal.connectInner (surreal.ts:93:20)
    at Surreal.connect (surreal.ts:84:22)
    at fetchData (ThoughtGalaxy.tsx:76:16)

2. Manual Calculation Required: Users had to manually click "Calculate My Graph" button to trigger UMAP dimensionality reduction

3. "Not enough nodes" despite having 3+ nodes: System was reporting insufficient nodes even after creating 3+ nodes with content

Root Causes

1. Client-Side Database Connection

The ThoughtGalaxy.tsx client component was attempting to connect directly to SurrealDB:

// ❌ Wrong: Client component trying to connect to database
import Surreal from 'surrealdb';

useEffect(() => {
  const db = new Surreal();
  await db.connect(process.env.NEXT_PUBLIC_SURREALDB_WSS_URL!); // undefined!
  // ...
}, []);

Problems:

• NEXT_PUBLIC_SURREALDB_WSS_URL environment variable didn't exist
• Client components shouldn't connect directly to databases (security/architecture violation)
• No authentication handling on client side

2. Manual Trigger Required

Graph calculation only happened when user clicked a button. No automatic detection of when calculation was needed.

3. Connection Method Inconsistency

The calculate-graph route was using inline database connection instead of the shared connectToDB() helper, leading to potential authentication mismatches.

Solutions

1. Created Server-Side Galaxy API Route

Created /app/api/galaxy/route.ts to handle all database access server-side:

export async function GET(request: NextRequest) {
  const cookieStore = await cookies();
  const surrealJwt = cookieStore.get('ponderants-auth')?.value;

  if (!surrealJwt) {
    return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
  }

  const userSession = verifySurrealJwt(surrealJwt);
  if (!userSession) {
    return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
  }

  const { did: userDid } = userSession;

  try {
    const db = await connectToDB();

    // Fetch nodes that have 3D coordinates
    const nodesQuery = `
      SELECT id, title, coords_3d
      FROM node
      WHERE user_did = $userDid AND coords_3d != NONE
    `;
    const nodeResults = await db.query<[NodeData[]]>(nodesQuery, { userDid });
    const nodes = nodeResults[0] || [];

    // Fetch links between nodes
    const linksQuery = `
      SELECT in, out
      FROM links_to
    `;
    const linkResults = await db.query<[LinkData[]]>(linksQuery);
    const links = linkResults[0] || [];

    // Auto-trigger calculation if needed
    if (nodes.length === 0) {
      const unmappedQuery = `
        SELECT count() as count
        FROM node
        WHERE user_did = $userDid AND embedding != NONE AND coords_3d = NONE
        GROUP ALL
      `;
      const unmappedResults = await db.query<[Array<{ count: number }>]>(unmappedQuery, { userDid });
      const unmappedCount = unmappedResults[0]?.[0]?.count || 0;

      if (unmappedCount >= 3) {
        console.log(`[Galaxy API] Found ${unmappedCount} unmapped nodes, triggering calculation...`);

        // Trigger graph calculation (don't await, return current state)
        fetch(`${process.env.NEXT_PUBLIC_BASE_URL || 'http://localhost:3000'}/api/calculate-graph`, {
          method: 'POST',
          headers: {
            'Cookie': `ponderants-auth=${surrealJwt}`,
          },
        }).catch((err) => {
          console.error('[Galaxy API] Failed to trigger graph calculation:', err);
        });

        return NextResponse.json({
          nodes: [],
          links: [],
          message: 'Calculating 3D coordinates... Refresh in a moment.',
        });
      }
    }

    console.log(`[Galaxy API] Returning ${nodes.length} nodes and ${links.length} links`);

    return NextResponse.json({
      nodes,
      links,
    });
  } catch (error) {
    console.error('[Galaxy API] Error:', error);
    return NextResponse.json(
      { error: 'Failed to fetch galaxy data' },
      { status: 500 }
    );
  }
}

Key features:

• ✅ Server-side authentication with JWT verification
• ✅ Data isolation via user_did filtering
• ✅ Auto-detection of unmapped nodes
• ✅ Automatic triggering of UMAP calculation
• ✅ Progress messaging for client polling

2. Updated ThoughtGalaxy Component

Changed from direct database connection to API-based data fetching:

Before:

import Surreal from 'surrealdb';

useEffect(() => {
  async function fetchData() {
    const db = new Surreal();
    await db.connect(process.env.NEXT_PUBLIC_SURREALDB_WSS_URL!);
    const token = document.cookie.split('ponderants-auth=')[1];
    await db.authenticate(token);

    const nodeResults = await db.query('SELECT id, title, coords_3d FROM node...');
    setNodes(nodeResults[0] || []);
  }
  fetchData();
}, []);

After:

// No Surreal import needed

useEffect(() => {
  async function fetchData() {
    try {
      const response = await fetch('/api/galaxy', {
        credentials: 'include', // Include cookies for authentication
      });

      if (!response.ok) {
        console.error('[ThoughtGalaxy] Failed to fetch galaxy data:', response.statusText);
        return;
      }

      const data = await response.json();

      if (data.message) {
        console.log('[ThoughtGalaxy]', data.message);
        // If calculating, poll again in 2 seconds
        setTimeout(fetchData, 2000);
        return;
      }

      setNodes(data.nodes || []);
      setLinks(data.links || []);

      console.log(`[ThoughtGalaxy] Loaded ${data.nodes?.length || 0} nodes and ${data.links?.length || 0} links`);
    } catch (error) {
      console.error('[ThoughtGalaxy] Error fetching data:', error);
    }
  }

  fetchData();
}, []);

Key improvements:

• ✅ No client-side database connection
• ✅ Proper authentication via HTTP-only cookies
• ✅ Polling mechanism for in-progress calculations
• ✅ Better error handling

3. Fixed calculate-graph Route

Updated /app/api/calculate-graph/route.ts to use shared helpers:

Before:

const db = new (await import('surrealdb')).default();
await db.connect(process.env.SURREALDB_URL!);
await db.signin({
  username: process.env.SURREALDB_USER!,
  password: process.env.SURREALDB_PASS!,
});

const jwt = require('jsonwebtoken');
const decoded = jwt.decode(surrealJwt) as { did: string };
const userDid = decoded?.did;

After:

import { connectToDB } from '@/lib/db';
import { verifySurrealJwt } from '@/lib/auth/jwt';

const userSession = verifySurrealJwt(surrealJwt);
if (!userSession) {
  return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
}

const { did: userDid } = userSession;

const db = await connectToDB();

Benefits:

• ✅ Consistent authentication across all routes
• ✅ Proper JWT verification (not just decode)
• ✅ Reusable code (DRY principle)

4. Created Debug Endpoint

Added /app/api/debug/nodes/route.ts for database inspection:

export async function GET(request: NextRequest) {
  const cookieStore = await cookies();
  const surrealJwt = cookieStore.get('ponderants-auth')?.value;

  if (!surrealJwt) {
    return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
  }

  const userSession = verifySurrealJwt(surrealJwt);
  if (!userSession) {
    return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
  }

  const { did: userDid } = userSession;

  try {
    const db = await connectToDB();

    const nodesQuery = `
      SELECT id, title, body, atp_uri, embedding, coords_3d
      FROM node
      WHERE user_did = $userDid
    `;
    const results = await db.query(nodesQuery, { userDid });
    const nodes = results[0] || [];

    const stats = {
      total: nodes.length,
      with_embeddings: nodes.filter((n: any) => n.embedding).length,
      with_coords: nodes.filter((n: any) => n.coords_3d).length,
      without_embeddings: nodes.filter((n: any) => !n.embedding).length,
      without_coords: nodes.filter((n: any) => !n.coords_3d).length,
    };

    return NextResponse.json({
      stats,
      nodes: nodes.map((n: any) => ({
        id: n.id,
        title: n.title,
        atp_uri: n.atp_uri,
        has_embedding: !!n.embedding,
        has_coords: !!n.coords_3d,
        coords_3d: n.coords_3d,
      })),
    });
  } catch (error) {
    console.error('[Debug Nodes] Error:', error);
    return NextResponse.json({ error: String(error) }, { status: 500 });
  }
}

Use: Visit /api/debug/nodes while logged in to see your node statistics and data.

Auto-Calculation Flow

1. User visits Galaxy page → ThoughtGalaxy component mounts
2. Component fetches data → GET /api/galaxy
3. API checks for coords → Query: WHERE coords_3d != NONE
4. If no coords found → Query unmapped count: WHERE embedding != NONE AND coords_3d = NONE
5. If ≥3 unmapped nodes → Trigger POST /api/calculate-graph (don't wait)
6. Return progress message → { message: 'Calculating 3D coordinates...' }
7. Client polls → setTimeout 2 seconds, fetch again
8. UMAP completes → Next poll returns actual node data
9. Client renders → 3D visualization appears

Files Changed

1. /components/ThoughtGalaxy.tsx - Removed direct DB connection, added API-based fetching and polling
2. /app/api/galaxy/route.ts - NEW - Server-side galaxy data endpoint with auto-calculation
3. /app/api/calculate-graph/route.ts - Updated to use connectToDB() and verifySurrealJwt()
4. /app/api/debug/nodes/route.ts - NEW - Debug endpoint for inspecting node data

Verification

After the fix:

# Server logs show auto-calculation:
[Galaxy API] Found 5 unmapped nodes, triggering calculation...
[Calculate Graph] Processing 5 nodes for UMAP projection
[Calculate Graph] Running UMAP dimensionality reduction...
[Calculate Graph] ✓ UMAP projection complete
[Calculate Graph] ✓ Updated 5 nodes with 3D coordinates

# Client logs show polling:
[ThoughtGalaxy] Calculating 3D coordinates... Refresh in a moment.
[ThoughtGalaxy] Loaded 5 nodes and 3 links

Architecture Note

This fix maintains the "Source of Truth vs. App View Cache" pattern:

• ATproto PDS - Canonical source of Node content (com.ponderants.node records)
• SurrealDB - Performance cache that stores:
  • Copy of node data for fast access
  • Vector embeddings for similarity search
  • Pre-computed 3D coordinates for visualization
  • Graph links between nodes

The auto-calculation ensures the cache stays enriched with visualization data without user intervention.