app/docs/fixes/galaxy-graph-fix.md

# Galaxy Graph Visualization Fix

## Problems

1. **Invalid URL Error**: ThoughtGalaxy component was failing with:
```
TypeError: Failed to construct 'URL': Invalid URL
    at parseUrl (surreal.ts:745:14)
    at Surreal.connectInner (surreal.ts:93:20)
    at Surreal.connect (surreal.ts:84:22)
    at fetchData (ThoughtGalaxy.tsx:76:16)
```

2. **Manual Calculation Required**: Users had to manually click "Calculate My Graph" button to trigger UMAP dimensionality reduction

3. **"Not enough nodes" despite having 3+ nodes**: System was reporting insufficient nodes even after creating 3+ nodes with content

## Root Causes

### 1. Client-Side Database Connection
The `ThoughtGalaxy.tsx` client component was attempting to connect directly to SurrealDB:

```typescript
// ❌ Wrong: Client component trying to connect to database
import Surreal from 'surrealdb';

useEffect(() => {
  const db = new Surreal();
  await db.connect(process.env.NEXT_PUBLIC_SURREALDB_WSS_URL!); // undefined!
  // ...
}, []);
```

Problems:
- `NEXT_PUBLIC_SURREALDB_WSS_URL` environment variable didn't exist
- Client components shouldn't connect directly to databases (security/architecture violation)
- No authentication handling on client side

### 2. Manual Trigger Required
Graph calculation only happened when user clicked a button. No automatic detection of when calculation was needed.

### 3. Connection Method Inconsistency
The `calculate-graph` route was using inline database connection instead of the shared `connectToDB()` helper, leading to potential authentication mismatches.

## Solutions

### 1. Created Server-Side Galaxy API Route
Created `/app/api/galaxy/route.ts` to handle all database access server-side:

```typescript
export async function GET(request: NextRequest) {
  const cookieStore = await cookies();
  const surrealJwt = cookieStore.get('ponderants-auth')?.value;

  if (!surrealJwt) {
    return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
  }

  const userSession = verifySurrealJwt(surrealJwt);
  if (!userSession) {
    return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
  }

  const { did: userDid } = userSession;

  try {
    const db = await connectToDB();

    // Fetch nodes that have 3D coordinates
    const nodesQuery = `
      SELECT id, title, coords_3d
      FROM node
      WHERE user_did = $userDid AND coords_3d != NONE
    `;
    const nodeResults = await db.query<[NodeData[]]>(nodesQuery, { userDid });
    const nodes = nodeResults[0] || [];

    // Fetch links between nodes
    const linksQuery = `
      SELECT in, out
      FROM links_to
    `;
    const linkResults = await db.query<[LinkData[]]>(linksQuery);
    const links = linkResults[0] || [];

    // Auto-trigger calculation if needed
    if (nodes.length === 0) {
      const unmappedQuery = `
        SELECT count() as count
        FROM node
        WHERE user_did = $userDid AND embedding != NONE AND coords_3d = NONE
        GROUP ALL
      `;
      const unmappedResults = await db.query<[Array<{ count: number }>]>(unmappedQuery, { userDid });
      const unmappedCount = unmappedResults[0]?.[0]?.count || 0;

      if (unmappedCount >= 3) {
        console.log(`[Galaxy API] Found ${unmappedCount} unmapped nodes, triggering calculation...`);

        // Trigger graph calculation (don't await, return current state)
        fetch(`${process.env.NEXT_PUBLIC_BASE_URL || 'http://localhost:3000'}/api/calculate-graph`, {
          method: 'POST',
          headers: {
            'Cookie': `ponderants-auth=${surrealJwt}`,
          },
        }).catch((err) => {
          console.error('[Galaxy API] Failed to trigger graph calculation:', err);
        });

        return NextResponse.json({
          nodes: [],
          links: [],
          message: 'Calculating 3D coordinates... Refresh in a moment.',
        });
      }
    }

    console.log(`[Galaxy API] Returning ${nodes.length} nodes and ${links.length} links`);

    return NextResponse.json({
      nodes,
      links,
    });
  } catch (error) {
    console.error('[Galaxy API] Error:', error);
    return NextResponse.json(
      { error: 'Failed to fetch galaxy data' },
      { status: 500 }
    );
  }
}
```

Key features:
- ✅ Server-side authentication with JWT verification
- ✅ Data isolation via `user_did` filtering
- ✅ Auto-detection of unmapped nodes
- ✅ Automatic triggering of UMAP calculation
- ✅ Progress messaging for client polling

### 2. Updated ThoughtGalaxy Component
Changed from direct database connection to API-based data fetching:

**Before:**
```typescript
import Surreal from 'surrealdb';

useEffect(() => {
  async function fetchData() {
    const db = new Surreal();
    await db.connect(process.env.NEXT_PUBLIC_SURREALDB_WSS_URL!);
    const token = document.cookie.split('ponderants-auth=')[1];
    await db.authenticate(token);

    const nodeResults = await db.query('SELECT id, title, coords_3d FROM node...');
    setNodes(nodeResults[0] || []);
  }
  fetchData();
}, []);
```

**After:**
```typescript
// No Surreal import needed

useEffect(() => {
  async function fetchData() {
    try {
      const response = await fetch('/api/galaxy', {
        credentials: 'include', // Include cookies for authentication
      });

      if (!response.ok) {
        console.error('[ThoughtGalaxy] Failed to fetch galaxy data:', response.statusText);
        return;
      }

      const data = await response.json();

      if (data.message) {
        console.log('[ThoughtGalaxy]', data.message);
        // If calculating, poll again in 2 seconds
        setTimeout(fetchData, 2000);
        return;
      }

      setNodes(data.nodes || []);
      setLinks(data.links || []);

      console.log(`[ThoughtGalaxy] Loaded ${data.nodes?.length || 0} nodes and ${data.links?.length || 0} links`);
    } catch (error) {
      console.error('[ThoughtGalaxy] Error fetching data:', error);
    }
  }

  fetchData();
}, []);
```

Key improvements:
- ✅ No client-side database connection
- ✅ Proper authentication via HTTP-only cookies
- ✅ Polling mechanism for in-progress calculations
- ✅ Better error handling

### 3. Fixed calculate-graph Route
Updated `/app/api/calculate-graph/route.ts` to use shared helpers:

**Before:**
```typescript
const db = new (await import('surrealdb')).default();
await db.connect(process.env.SURREALDB_URL!);
await db.signin({
  username: process.env.SURREALDB_USER!,
  password: process.env.SURREALDB_PASS!,
});

const jwt = require('jsonwebtoken');
const decoded = jwt.decode(surrealJwt) as { did: string };
const userDid = decoded?.did;
```

**After:**
```typescript
import { connectToDB } from '@/lib/db';
import { verifySurrealJwt } from '@/lib/auth/jwt';

const userSession = verifySurrealJwt(surrealJwt);
if (!userSession) {
  return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
}

const { did: userDid } = userSession;

const db = await connectToDB();
```

Benefits:
- ✅ Consistent authentication across all routes
- ✅ Proper JWT verification (not just decode)
- ✅ Reusable code (DRY principle)

### 4. Created Debug Endpoint
Added `/app/api/debug/nodes/route.ts` for database inspection:

```typescript
export async function GET(request: NextRequest) {
  const cookieStore = await cookies();
  const surrealJwt = cookieStore.get('ponderants-auth')?.value;

  if (!surrealJwt) {
    return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
  }

  const userSession = verifySurrealJwt(surrealJwt);
  if (!userSession) {
    return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
  }

  const { did: userDid } = userSession;

  try {
    const db = await connectToDB();

    const nodesQuery = `
      SELECT id, title, body, atp_uri, embedding, coords_3d
      FROM node
      WHERE user_did = $userDid
    `;
    const results = await db.query(nodesQuery, { userDid });
    const nodes = results[0] || [];

    const stats = {
      total: nodes.length,
      with_embeddings: nodes.filter((n: any) => n.embedding).length,
      with_coords: nodes.filter((n: any) => n.coords_3d).length,
      without_embeddings: nodes.filter((n: any) => !n.embedding).length,
      without_coords: nodes.filter((n: any) => !n.coords_3d).length,
    };

    return NextResponse.json({
      stats,
      nodes: nodes.map((n: any) => ({
        id: n.id,
        title: n.title,
        atp_uri: n.atp_uri,
        has_embedding: !!n.embedding,
        has_coords: !!n.coords_3d,
        coords_3d: n.coords_3d,
      })),
    });
  } catch (error) {
    console.error('[Debug Nodes] Error:', error);
    return NextResponse.json({ error: String(error) }, { status: 500 });
  }
}
```

Use: Visit `/api/debug/nodes` while logged in to see your node statistics and data.

## Auto-Calculation Flow

1. **User visits Galaxy page** → ThoughtGalaxy component mounts
2. **Component fetches data** → `GET /api/galaxy`
3. **API checks for coords** → Query: `WHERE coords_3d != NONE`
4. **If no coords found** → Query unmapped count: `WHERE embedding != NONE AND coords_3d = NONE`
5. **If ≥3 unmapped nodes** → Trigger `POST /api/calculate-graph` (don't wait)
6. **Return progress message** → `{ message: 'Calculating 3D coordinates...' }`
7. **Client polls** → setTimeout 2 seconds, fetch again
8. **UMAP completes** → Next poll returns actual node data
9. **Client renders** → 3D visualization appears

## Files Changed

1. `/components/ThoughtGalaxy.tsx` - Removed direct DB connection, added API-based fetching and polling
2. `/app/api/galaxy/route.ts` - **NEW** - Server-side galaxy data endpoint with auto-calculation
3. `/app/api/calculate-graph/route.ts` - Updated to use `connectToDB()` and `verifySurrealJwt()`
4. `/app/api/debug/nodes/route.ts` - **NEW** - Debug endpoint for inspecting node data

## Verification

After the fix:

```bash
# Server logs show auto-calculation:
[Galaxy API] Found 5 unmapped nodes, triggering calculation...
[Calculate Graph] Processing 5 nodes for UMAP projection
[Calculate Graph] Running UMAP dimensionality reduction...
[Calculate Graph] ✓ UMAP projection complete
[Calculate Graph] ✓ Updated 5 nodes with 3D coordinates
```

```bash
# Client logs show polling:
[ThoughtGalaxy] Calculating 3D coordinates... Refresh in a moment.
[ThoughtGalaxy] Loaded 5 nodes and 3 links
```

## Architecture Note

This fix maintains the "Source of Truth vs. App View Cache" pattern:
- **ATproto PDS** - Canonical source of Node content (com.ponderants.node records)
- **SurrealDB** - Performance cache that stores:
  - Copy of node data for fast access
  - Vector embeddings for similarity search
  - Pre-computed 3D coordinates for visualization
  - Graph links between nodes

The auto-calculation ensures the cache stays enriched with visualization data without user intervention.