app/docs/umap-recalculation-strategy.md

UMAP Recalculation Strategy

Problem Statement

When creating the 3D thought galaxy visualization, we need to convert high-dimensional AI embeddings (3072 dimensions from gemini-embedding-001) into 3D coordinates that can be displayed in the browser.

The Challenge

Question: Should we calculate coordinates incrementally (one node at a time) or recalculate ALL nodes together every time?

Initial broken approach:

-- Only calculate for nodes without coordinates
SELECT id, embedding FROM node
WHERE user_did = $userDid
  AND embedding != NONE
  AND coords_3d = NONE

This caused a bug where:

1. Nodes 1-3: Calculate together → ✓ Get coords
2. Nodes 4-5: Try to calculate separately → ✗ FAILS (only 2 points, UMAP needs 3+)

Why UMAP Requires Recalculation

What is UMAP?

UMAP (Uniform Manifold Approximation and Projection) is a non-linear manifold learning algorithm. Unlike linear methods (PCA), UMAP:

1. Learns the "shape" (manifold) of your data - It finds clusters, relationships, and patterns
2. Creates relative, not absolute coordinates - There's no fixed origin or coordinate system
3. Requires seeing all data together - The manifold structure changes as you add more data

Why Incremental Doesn't Work

Problem with fixed origin approach:

# Each run produces DIFFERENT coordinates!
Run 1: UMAP([node1, node2, node3]) → coords_A
Run 2: UMAP([node1, node2, node3]) → coords_B  # DIFFERENT!

# There's no absolute coordinate system
Run 1: node1 at [0.5, 0.2, 0.8]
Run 2: node1 at [2.1, -1.3, 0.4]  # Completely different!

The positions are only meaningful relative to each other. You can't have a "fixed origin" because UMAP learns a relative manifold structure.

Why you need 3+ points:

• UMAP is a manifold learning algorithm
• A manifold requires multiple points to define a shape
• With only 1-2 points, there's no "manifold" to learn

What About UMAP.transform()?

UMAP does support an incremental transform() method:

# Fit once, save the model
umap_model = UMAP(n_components=3)
umap_model.fit(initial_embeddings)

# Transform new points into existing space
new_coords = umap_model.transform(new_embedding)

Why we're NOT using this:

1. Model storage complexity - Must store entire UMAP model (includes all training data) in database
2. Model drift - New nodes get approximate positions based on old manifold structure
3. Loss of quality - The manifold changes as you add data; transform() doesn't update it
4. Performance - For <100 nodes, full recalculation is fast (<1 second)

Our Solution: Full Recalculation

Implementation

-- Recalculate ALL nodes every time
SELECT id, embedding FROM node
WHERE user_did = $userDid
  AND embedding != NONE
-- No "coords_3d = NONE" filter!

Behavior

When you add a new node:

1. Fetch ALL nodes with embeddings (including those with existing coords)
2. Run UMAP on the complete dataset
3. Update ALL nodes with their recalculated positions

Result: The galaxy "reorganizes" when you add new thoughts - existing nodes WILL move slightly.

Trade-offs

Pros: ✅ Always mathematically correct ✅ Simple implementation ✅ No model storage complexity ✅ Best clustering quality (manifold adapts to new data) ✅ Fast enough for <100 nodes

Cons: ❌ Galaxy shifts when adding nodes (existing nodes move) ❌ O(n²) complexity (slower with many nodes) ❌ More database writes

Performance Characteristics

Nodes | Calculation Time | Acceptable?
------|-----------------|------------
3     | ~50ms           | ✅ Excellent
10    | ~200ms          | ✅ Great
50    | ~800ms          | ✅ Good
100   | ~1.5s           | ✅ Acceptable
500   | ~15s            | ⚠️ Slow (consider optimization)
1000+ | ~60s+           | ❌ Too slow (need incremental)

For the Ponderants MVP, we expect users to have <100 nodes, making full recalculation perfectly acceptable.

Future Optimizations

If we reach scale where recalculation becomes too slow:

Option 1: UMAP.transform() with Periodic Refitting

// Store UMAP model in database
// Transform new nodes incrementally
// Every 10 nodes: Refit the entire model
if (newNodeCount % 10 === 0) {
  recalculateAllNodes();
}

Option 2: Switch to PCA

• PCA is linear and supports incremental updates
• Loses UMAP's superior clustering quality
• Use for very large datasets (1000+ nodes)

Option 3: Hierarchical UMAP

• Cluster nodes into groups
• Run UMAP on each cluster separately
• Use a higher-level UMAP to arrange clusters
• Complex but scales to millions of nodes

User Experience

The galaxy "reorganizing" when you add nodes is actually a feature, not a bug:

• It shows your thought network evolving
• New connections emerge as you add ideas
• Clusters naturally form around related concepts
• Creates a sense of a living, breathing knowledge graph

Users will see their constellation of thoughts naturally reorganize as their ideas grow - which aligns perfectly with the "Ponderants" brand of exploring and structuring ideas.

References

• UMAP Documentation
• umap-js Library
• Understanding UMAP
• When to use UMAP vs PCA

Decision Log

• 2025-01-10: Discovered bug where nodes 4-5 failed to get coordinates
• 2025-01-10: Analyzed UMAP manifold learning constraints
• 2025-01-10: Decided to implement full recalculation strategy
• 2025-01-10: Updated /app/api/calculate-graph/route.ts to remove coords_3d = NONE filter