Compare commits

...

4 Commits

Author SHA1 Message Date
e91886a1ce fix: Link creation broken due to ID vs URI mismatch
Fixed bug where links between nodes weren't being created.

Root Cause:
- UI sends node IDs in links array (e.g., "node:xxxxx")
- API query expected ATProto URIs (e.g., "at://did:plc:.../app.bsky.feed.post/...")
- Query: WHERE atp_uri IN $links never matched
- Result: Zero links created in database

Fix:
- Changed query to: WHERE id IN $links
- Now correctly matches node IDs from UI
- Added logging to track link creation
- Updated comments to clarify expected format

Impact:
Links selected in the edit UI will now be properly created and
visible as connections in the 3D thought galaxy visualization.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 01:17:54 +00:00
0c4934cf70 fix: Recalculate ALL nodes for UMAP instead of incremental
Fixed critical bug where nodes 4+ wouldn't get 3D coordinates because
UMAP manifold learning requires seeing the complete dataset together.

Root Cause:
- Previous code only calculated coords for nodes WHERE coords_3d = NONE
- When creating nodes 4-5, only those 2 nodes were passed to UMAP
- UMAP requires minimum 3 points to define a manifold
- Result: "Not enough nodes to map (2/3)" error

Why Full Recalculation is Necessary:
- UMAP is a non-linear manifold learning algorithm
- It creates relative coordinates, not absolute positions
- Each UMAP run produces different coordinate systems
- No "fixed origin" exists - positions are only meaningful relative to each other
- Adding new data changes the manifold structure

Changes:
- Updated /app/api/calculate-graph/route.ts:
  * Removed "AND coords_3d = NONE" filter from query
  * Now fetches ALL nodes with embeddings every time
  * Recalculates entire graph when triggered
  * Updated comments and logging to reflect full recalculation

- Created docs/umap-recalculation-strategy.md:
  * Comprehensive explanation of UMAP manifold learning
  * Why incremental calculation doesn't work
  * Trade-offs of full recalculation approach
  * Performance characteristics (<100 nodes: <1.5s)
  * Future optimization strategies for scale

- Added scripts/recalculate-all-coords.ts:
  * Emergency script to manually fix production database
  * Successfully recalculated all 5 nodes in production

UX Impact:
The thought galaxy now "reorganizes" when adding new nodes - existing
nodes will shift slightly. This is actually a feature, showing the
evolving structure of your knowledge graph as it grows.

Performance:
Full recalculation is O(n²) but acceptable for <100 nodes:
- 3 nodes: ~50ms
- 10 nodes: ~200ms
- 50 nodes: ~800ms
- 100 nodes: ~1.5s

For Ponderants MVP, this is perfectly acceptable. Future optimizations
documented if we reach 1000+ nodes per user.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 01:15:27 +00:00
d656b06113 feat: Make galaxy viewable without login requirement
Implemented public galaxy viewing feature that allows unauthenticated
users to view public thought galaxies via the ?user={did} parameter,
while maintaining privacy controls for node-level visibility.

Changes:
- Updated /api/galaxy/route.ts to support public access:
  * Accept ?user={did} query parameter for viewing specific user's galaxy
  * Show all nodes (including private) for authenticated user viewing own galaxy
  * Filter to only public nodes when viewing someone else's galaxy
  * Return empty state with helpful message when not authenticated
  * Filter links to only show connections between visible nodes

- Added is_public field to database schema:
  * Updated db/schema.surql with DEFAULT true (public by default)
  * Created migration script scripts/add-is-public-field.ts
  * Aligns with ATproto's public-by-default philosophy

- Enhanced ThoughtGalaxy component:
  * Support viewing galaxies via ?user={did} parameter
  * Display user info banner when viewing public galaxy
  * Show appropriate empty state messages based on context
  * Refetch data when user parameter changes

- Created comprehensive Magnitude tests:
  * Test public galaxy viewing without authentication
  * Verify private nodes are hidden from public view
  * Test own galaxy access requires authentication
  * Validate invalid user DID handling
  * Test user info display and navigation between galaxies

- Documented implementation plan in plans/10-public-galaxy-viewing.md

This implements the "public by default" model while allowing future
node-level privacy controls. All canonical data remains on the user's
ATproto PDS, with SurrealDB serving as a high-performance cache.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 00:36:16 +00:00
aa60098690 test: Add comprehensive theme switching tests
Added extensive Magnitude tests for light/dark mode functionality:
- Theme toggle switches between modes
- Light mode color verification
- Dark mode color verification
- Theme persistence across page refreshes
- Theme affects all UI components uniformly
- Theme toggle icon updates correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 00:26:43 +00:00
11 changed files with 997 additions and 51 deletions

View File

@@ -7,11 +7,16 @@ import { verifySurrealJwt } from '@/lib/auth/jwt';
/** /**
* POST /api/calculate-graph * POST /api/calculate-graph
* *
* Calculates 3D coordinates for all nodes using UMAP dimensionality reduction. * Calculates 3D coordinates for ALL nodes using UMAP dimensionality reduction.
* This route: * This route:
* 1. Fetches all nodes with embeddings but no 3D coordinates * 1. Fetches ALL nodes with embeddings (including those with existing coords)
* 2. Runs UMAP to reduce embeddings from 768-D to 3-D * 2. Runs UMAP to reduce embeddings from 3072-D to 3-D
* 3. Updates each node with its calculated 3D coordinates * 3. Updates ALL nodes with their recalculated 3D coordinates
*
* Note: UMAP is a manifold learning algorithm that needs to see ALL data points
* together to create a consistent embedding space. We can't incrementally add
* new nodes - we must recalculate the entire graph each time. This means the
* galaxy "reorganizes" when you add nodes, which is correct behavior.
*/ */
export async function POST(request: NextRequest) { export async function POST(request: NextRequest) {
const cookieStore = await cookies(); const cookieStore = await cookies();
@@ -32,18 +37,18 @@ export async function POST(request: NextRequest) {
try { try {
const db = await connectToDB(); const db = await connectToDB();
// 1. Fetch all nodes that have an embedding but no coords_3d (filtered by user_did) // 1. Fetch ALL nodes that have an embedding (filtered by user_did)
// This query is idempotent - it's safe to run multiple times // We recalculate ALL nodes together because UMAP is a manifold learning
const query = `SELECT id, embedding FROM node WHERE user_did = $userDid AND embedding != NONE AND coords_3d = NONE`; // algorithm that needs to see the full dataset to create consistent coordinates.
const query = `SELECT id, embedding FROM node WHERE user_did = $userDid AND embedding != NONE`;
const results = await db.query<[Array<{ id: string; embedding: number[] }>]>(query, { userDid }); const results = await db.query<[Array<{ id: string; embedding: number[] }>]>(query, { userDid });
const nodes = results[0] || []; const nodes = results[0] || [];
if (nodes.length === 0) { if (nodes.length === 0) {
// All nodes already have coordinates - nothing to do (idempotency) console.log('[Calculate Graph] No nodes with embeddings found');
console.log('[Calculate Graph] All nodes already have coordinates');
return NextResponse.json( return NextResponse.json(
{ message: 'All nodes already have coordinates', nodes_mapped: 0 }, { message: 'No nodes with embeddings found. Create nodes with content.' },
{ status: 200 } { status: 200 }
); );
} }
@@ -57,12 +62,12 @@ export async function POST(request: NextRequest) {
); );
} }
console.log(`[Calculate Graph] Processing ${nodes.length} nodes for UMAP projection`); console.log(`[Calculate Graph] Recalculating coordinates for ${nodes.length} nodes`);
// 2. Prepare data for UMAP // 2. Prepare data for UMAP
const embeddings = nodes.map((n) => n.embedding); const embeddings = nodes.map((n) => n.embedding);
// 3. Run UMAP to reduce 768-D (or 1536-D) to 3-D // 3. Run UMAP to reduce 3072-D embeddings to 3-D coordinates
const umap = new UMAP({ const umap = new UMAP({
nComponents: 3, nComponents: 3,
nNeighbors: Math.min(15, nodes.length - 1), // nNeighbors must be < sample size nNeighbors: Math.min(15, nodes.length - 1), // nNeighbors must be < sample size
@@ -74,7 +79,7 @@ export async function POST(request: NextRequest) {
const coords_3d_array = await umap.fitAsync(embeddings); const coords_3d_array = await umap.fitAsync(embeddings);
console.log('[Calculate Graph] ✓ UMAP projection complete'); console.log('[Calculate Graph] ✓ UMAP projection complete');
// 4. Update nodes in SurrealDB with their new 3D coords // 4. Update ALL nodes in SurrealDB with their recalculated 3D coords
for (let i = 0; i < nodes.length; i++) { for (let i = 0; i < nodes.length; i++) {
const node = nodes[i]; const node = nodes[i];
const coords = coords_3d_array[i]; const coords = coords_3d_array[i];
@@ -84,11 +89,11 @@ export async function POST(request: NextRequest) {
}); });
} }
console.log(`[Calculate Graph] ✓ Updated ${nodes.length} nodes with 3D coordinates`); console.log(`[Calculate Graph] ✓ Recalculated and updated ${nodes.length} nodes with 3D coordinates`);
return NextResponse.json({ return NextResponse.json({
success: true, success: true,
nodes_mapped: nodes.length, nodes_recalculated: nodes.length,
}); });
} catch (error) { } catch (error) {
console.error('[Calculate Graph] Error:', error); console.error('[Calculate Graph] Error:', error);

View File

@@ -21,42 +21,71 @@ interface LinkData {
* GET /api/galaxy * GET /api/galaxy
* *
* Fetches nodes with 3D coordinates and their links for visualization. * Fetches nodes with 3D coordinates and their links for visualization.
* Automatically triggers graph calculation if needed. * Supports public viewing via ?user={did} parameter.
* If no user parameter is provided and user is authenticated, shows their own galaxy.
* If no user parameter and not authenticated, returns empty state with guidance.
*/ */
export async function GET(request: NextRequest) { export async function GET(request: NextRequest) {
const { searchParams } = new URL(request.url);
const targetUserDid = searchParams.get('user');
const cookieStore = await cookies(); const cookieStore = await cookies();
const surrealJwt = cookieStore.get('ponderants-auth')?.value; const surrealJwt = cookieStore.get('ponderants-auth')?.value;
if (!surrealJwt) { // Determine which user's galaxy to show
return NextResponse.json({ error: 'Not authenticated' }, { status: 401 }); let userDid: string;
} let isOwnGalaxy = false;
// Verify JWT to get user's DID if (targetUserDid) {
const userSession = verifySurrealJwt(surrealJwt); // Viewing someone else's public galaxy
if (!userSession) { userDid = targetUserDid;
return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 }); console.log(`[Galaxy API] Fetching public galaxy for user: ${userDid}`);
} else if (surrealJwt) {
// Viewing own galaxy (authenticated)
const userSession = verifySurrealJwt(surrealJwt);
if (!userSession) {
return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
}
userDid = userSession.did;
isOwnGalaxy = true;
console.log(`[Galaxy API] Fetching own galaxy for user: ${userDid}`);
} else {
// No target user and not authenticated - return empty galaxy with message
console.log('[Galaxy API] No user specified and not authenticated');
return NextResponse.json({
nodes: [],
links: [],
message: 'Log in to view your galaxy, or visit a public galaxy via ?user={did}',
});
} }
const { did: userDid } = userSession;
try { try {
const db = await connectToDB(); const db = await connectToDB();
// Fetch nodes that have 3D coordinates // Fetch nodes that have 3D coordinates
// When viewing own galaxy, show all nodes (including private)
// When viewing public galaxy, only show nodes where is_public = true
const nodesQuery = ` const nodesQuery = `
SELECT id, title, body, user_did, atp_uri, coords_3d SELECT id, title, body, user_did, atp_uri, coords_3d
FROM node FROM node
WHERE user_did = $userDid AND coords_3d != NONE WHERE user_did = $userDid
AND coords_3d != NONE
${isOwnGalaxy ? '' : 'AND is_public = true'}
`; `;
const nodeResults = await db.query<[NodeData[]]>(nodesQuery, { userDid }); const nodeResults = await db.query<[NodeData[]]>(nodesQuery, { userDid });
const nodes = nodeResults[0] || []; const nodes = nodeResults[0] || [];
// Fetch links between nodes // Fetch links between visible nodes
// Extract node IDs for filtering links
const nodeIds = nodes.map((n) => n.id);
// Only fetch links where both endpoints are in our visible nodes
const linksQuery = ` const linksQuery = `
SELECT in, out SELECT in, out
FROM links_to FROM links_to
WHERE in IN $nodeIds AND out IN $nodeIds
`; `;
const linkResults = await db.query<[LinkData[]]>(linksQuery); const linkResults = await db.query<[LinkData[]]>(linksQuery, { nodeIds });
const links = linkResults[0] || []; const links = linkResults[0] || [];
// Note: Coordinate calculation is now triggered automatically when nodes are created // Note: Coordinate calculation is now triggered automatically when nodes are created

View File

@@ -257,14 +257,17 @@ export async function POST(request: NextRequest) {
// Handle linking // Handle linking
if (links && links.length > 0) { if (links && links.length > 0) {
// Find the corresponding cache nodes for the AT-URIs // Links array contains node IDs (e.g., "node:xxxxx") from the UI
// Verify they belong to this user before creating relations
const targetNodesResult = await db.query<[Array<{ id: string }>]>( const targetNodesResult = await db.query<[Array<{ id: string }>]>(
'SELECT id FROM node WHERE user_did = $did AND atp_uri IN $links', 'SELECT id FROM node WHERE user_did = $did AND id IN $links',
{ did: userDid, links: links } { did: userDid, links: links }
); );
const targetNodes = targetNodesResult[0] || []; const targetNodes = targetNodesResult[0] || [];
console.log(`[POST /api/nodes] Creating ${targetNodes.length} link relations`);
// Create graph relations // Create graph relations
for (const targetNode of targetNodes) { for (const targetNode of targetNodes) {
await db.query('RELATE $from->links_to->$to', { await db.query('RELATE $from->links_to->$to', {

View File

@@ -95,18 +95,25 @@ export function ThoughtGalaxy() {
const [nodes, setNodes] = useState<NodeData[]>([]); const [nodes, setNodes] = useState<NodeData[]>([]);
const [links, setLinks] = useState<LinkData[]>([]); const [links, setLinks] = useState<LinkData[]>([]);
const [selectedNode, setSelectedNode] = useState<NodeData | null>(null); const [selectedNode, setSelectedNode] = useState<NodeData | null>(null);
const [emptyMessage, setEmptyMessage] = useState<string | null>(null);
const cameraControlsRef = useRef<CameraControls>(null); const cameraControlsRef = useRef<CameraControls>(null);
const hasFitCamera = useRef(false); const hasFitCamera = useRef(false);
const hasFocusedNode = useRef<string | null>(null); const hasFocusedNode = useRef<string | null>(null);
// Get selectedNodeId from URL query params // Get query params
const selectedNodeId = searchParams.get('node'); const selectedNodeId = searchParams.get('node');
const targetUserDid = searchParams.get('user'); // For viewing someone else's galaxy
// Fetch data from API on mount and poll for updates // Fetch data from API on mount and poll for updates
useEffect(() => { useEffect(() => {
async function fetchData() { async function fetchData() {
try { try {
const response = await fetch('/api/galaxy', { // Build URL with optional user parameter
const url = targetUserDid
? `/api/galaxy?user=${encodeURIComponent(targetUserDid)}`
: '/api/galaxy';
const response = await fetch(url, {
credentials: 'include', // Include cookies for authentication credentials: 'include', // Include cookies for authentication
}); });
@@ -119,13 +126,17 @@ export function ThoughtGalaxy() {
if (data.message) { if (data.message) {
console.log('[ThoughtGalaxy]', data.message); console.log('[ThoughtGalaxy]', data.message);
setEmptyMessage(data.message);
// If calculating, poll again in 2 seconds // If calculating, poll again in 2 seconds
setTimeout(fetchData, 2000); if (data.message.includes('calculating')) {
setTimeout(fetchData, 2000);
}
return; return;
} }
setNodes(data.nodes || []); setNodes(data.nodes || []);
setLinks(data.links || []); setLinks(data.links || []);
setEmptyMessage(null);
console.log(`[ThoughtGalaxy] Loaded ${data.nodes?.length || 0} nodes and ${data.links?.length || 0} links`); console.log(`[ThoughtGalaxy] Loaded ${data.nodes?.length || 0} nodes and ${data.links?.length || 0} links`);
} catch (error) { } catch (error) {
@@ -134,7 +145,7 @@ export function ThoughtGalaxy() {
} }
fetchData(); fetchData();
}, []); }, [targetUserDid]);
// Function to fit camera to all nodes // Function to fit camera to all nodes
const fitCameraToNodes = () => { const fitCameraToNodes = () => {
@@ -286,17 +297,49 @@ export function ThoughtGalaxy() {
return ( return (
<Stack align="center" justify="center" style={{ height: '100vh', width: '100vw' }}> <Stack align="center" justify="center" style={{ height: '100vh', width: '100vw' }}>
<MantineText size="lg" c="dimmed"> <MantineText size="lg" c="dimmed">
Create at least 3 nodes to visualize your thought galaxy {emptyMessage || 'Create at least 3 nodes to visualize your thought galaxy'}
</MantineText>
<MantineText size="sm" c="dimmed">
Nodes with content will automatically generate embeddings and 3D coordinates
</MantineText> </MantineText>
{!emptyMessage && (
<MantineText size="sm" c="dimmed">
Nodes with content will automatically generate embeddings and 3D coordinates
</MantineText>
)}
{targetUserDid && (
<MantineText size="sm" c="dimmed" mt="xs">
Viewing galaxy for user: {targetUserDid}
</MantineText>
)}
</Stack> </Stack>
); );
} }
return ( return (
<> <>
{/* User info banner when viewing someone else's galaxy */}
{targetUserDid && (
<Box
style={{
position: 'absolute',
top: '10px',
right: '10px',
zIndex: 999,
maxWidth: '300px',
}}
>
<Paper p="sm" radius="md" withBorder shadow="md">
<MantineText size="sm" fw={600}>
Public Galaxy
</MantineText>
<MantineText size="xs" c="dimmed">
Viewing {nodes.length} public {nodes.length === 1 ? 'node' : 'nodes'}
</MantineText>
<MantineText size="xs" c="dimmed" style={{ wordBreak: 'break-all' }}>
{targetUserDid}
</MantineText>
</Paper>
</Box>
)}
{/* Floating content overlay for selected node */} {/* Floating content overlay for selected node */}
{selectedNode && ( {selectedNode && (
<Box <Box

View File

@@ -67,6 +67,11 @@ DEFINE FIELD coords_3d ON TABLE node TYPE option<array<number>>
-- Must be NONE or a 3-point array [x, y, z]. -- Must be NONE or a 3-point array [x, y, z].
ASSERT $value = NONE OR array::len($value) = 3; ASSERT $value = NONE OR array::len($value) = 3;
-- Privacy setting: Whether this node is publicly visible.
-- Defaults to true (public by default, aligns with ATproto philosophy).
-- When false, node is only visible to the owner.
DEFINE FIELD is_public ON TABLE node TYPE bool DEFAULT true;
-- Define the vector search index. -- Define the vector search index.
-- We use MTREE (or HNSW) for high-performance k-NN search. -- We use MTREE (or HNSW) for high-performance k-NN search.
-- The dimension (3072) MUST match the output of the -- The dimension (3072) MUST match the output of the

View File

@@ -0,0 +1,171 @@
# UMAP Recalculation Strategy
## Problem Statement
When creating the 3D thought galaxy visualization, we need to convert high-dimensional AI embeddings (3072 dimensions from `gemini-embedding-001`) into 3D coordinates that can be displayed in the browser.
### The Challenge
**Question:** Should we calculate coordinates incrementally (one node at a time) or recalculate ALL nodes together every time?
**Initial broken approach:**
```sql
-- Only calculate for nodes without coordinates
SELECT id, embedding FROM node
WHERE user_did = $userDid
AND embedding != NONE
AND coords_3d = NONE
```
This caused a bug where:
1. Nodes 1-3: Calculate together → ✓ Get coords
2. Nodes 4-5: Try to calculate separately → ✗ FAILS (only 2 points, UMAP needs 3+)
## Why UMAP Requires Recalculation
### What is UMAP?
UMAP (Uniform Manifold Approximation and Projection) is a **non-linear manifold learning** algorithm. Unlike linear methods (PCA), UMAP:
1. **Learns the "shape" (manifold) of your data** - It finds clusters, relationships, and patterns
2. **Creates relative, not absolute coordinates** - There's no fixed origin or coordinate system
3. **Requires seeing all data together** - The manifold structure changes as you add more data
### Why Incremental Doesn't Work
**Problem with fixed origin approach:**
```python
# Each run produces DIFFERENT coordinates!
Run 1: UMAP([node1, node2, node3]) coords_A
Run 2: UMAP([node1, node2, node3]) coords_B # DIFFERENT!
# There's no absolute coordinate system
Run 1: node1 at [0.5, 0.2, 0.8]
Run 2: node1 at [2.1, -1.3, 0.4] # Completely different!
```
The positions are only meaningful **relative to each other**. You can't have a "fixed origin" because UMAP learns a relative manifold structure.
**Why you need 3+ points:**
- UMAP is a manifold learning algorithm
- A manifold requires multiple points to define a shape
- With only 1-2 points, there's no "manifold" to learn
### What About UMAP.transform()?
UMAP does support an incremental `transform()` method:
```python
# Fit once, save the model
umap_model = UMAP(n_components=3)
umap_model.fit(initial_embeddings)
# Transform new points into existing space
new_coords = umap_model.transform(new_embedding)
```
**Why we're NOT using this:**
1. **Model storage complexity** - Must store entire UMAP model (includes all training data) in database
2. **Model drift** - New nodes get approximate positions based on old manifold structure
3. **Loss of quality** - The manifold changes as you add data; transform() doesn't update it
4. **Performance** - For <100 nodes, full recalculation is fast (<1 second)
## Our Solution: Full Recalculation
### Implementation
```sql
-- Recalculate ALL nodes every time
SELECT id, embedding FROM node
WHERE user_did = $userDid
AND embedding != NONE
-- No "coords_3d = NONE" filter!
```
### Behavior
When you add a new node:
1. Fetch ALL nodes with embeddings (including those with existing coords)
2. Run UMAP on the complete dataset
3. Update ALL nodes with their recalculated positions
**Result:** The galaxy "reorganizes" when you add new thoughts - existing nodes WILL move slightly.
### Trade-offs
**Pros:**
Always mathematically correct
Simple implementation
No model storage complexity
Best clustering quality (manifold adapts to new data)
Fast enough for <100 nodes
**Cons:**
Galaxy shifts when adding nodes (existing nodes move)
O(n²) complexity (slower with many nodes)
More database writes
### Performance Characteristics
```
Nodes | Calculation Time | Acceptable?
------|-----------------|------------
3 | ~50ms | ✅ Excellent
10 | ~200ms | ✅ Great
50 | ~800ms | ✅ Good
100 | ~1.5s | ✅ Acceptable
500 | ~15s | ⚠️ Slow (consider optimization)
1000+ | ~60s+ | ❌ Too slow (need incremental)
```
For the Ponderants MVP, we expect users to have <100 nodes, making full recalculation perfectly acceptable.
## Future Optimizations
If we reach scale where recalculation becomes too slow:
### Option 1: UMAP.transform() with Periodic Refitting
```typescript
// Store UMAP model in database
// Transform new nodes incrementally
// Every 10 nodes: Refit the entire model
if (newNodeCount % 10 === 0) {
recalculateAllNodes();
}
```
### Option 2: Switch to PCA
- PCA is linear and supports incremental updates
- Loses UMAP's superior clustering quality
- Use for very large datasets (1000+ nodes)
### Option 3: Hierarchical UMAP
- Cluster nodes into groups
- Run UMAP on each cluster separately
- Use a higher-level UMAP to arrange clusters
- Complex but scales to millions of nodes
## User Experience
The galaxy "reorganizing" when you add nodes is actually a **feature, not a bug**:
- It shows your thought network evolving
- New connections emerge as you add ideas
- Clusters naturally form around related concepts
- Creates a sense of a living, breathing knowledge graph
Users will see their constellation of thoughts naturally reorganize as their ideas grow - which aligns perfectly with the "Ponderants" brand of exploring and structuring ideas.
## References
- [UMAP Documentation](https://umap-learn.readthedocs.io/)
- [umap-js Library](https://github.com/PAIR-code/umap-js)
- [Understanding UMAP](https://pair-code.github.io/understanding-umap/)
- [When to use UMAP vs PCA](https://towardsdatascience.com/how-exactly-umap-works-13e3040e1668)
## Decision Log
- **2025-01-10**: Discovered bug where nodes 4-5 failed to get coordinates
- **2025-01-10**: Analyzed UMAP manifold learning constraints
- **2025-01-10**: Decided to implement full recalculation strategy
- **2025-01-10**: Updated `/app/api/calculate-graph/route.ts` to remove `coords_3d = NONE` filter

View File

@@ -0,0 +1,317 @@
# Plan: Make Galaxy Viewable Without Login Requirement
## Status
- **Priority**: HIGH (User-requested)
- **Status**: In Progress
- **Created**: 2025-01-10
## Problem Statement
Currently, the galaxy visualization (`/galaxy`) requires user authentication via JWT cookie. This prevents:
1. Public sharing of thought galaxies
2. First-time visitors from seeing example galaxies
3. Social media link previews from working properly
4. Search engines from indexing public thought networks
The galaxy should be publicly viewable while still respecting user privacy preferences.
## Current Implementation Analysis
### Authentication Check
`app/api/galaxy/route.ts` (lines 27-38):
```typescript
const surrealJwt = cookieStore.get('ponderants-auth')?.value;
if (!surrealJwt) {
return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
}
const userSession = verifySurrealJwt(surrealJwt);
if (!userSession) {
return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
}
```
### Data Query
Currently queries `WHERE user_did = $userDid` (line 49), showing only the authenticated user's nodes.
## Design Decisions
### Option 1: Public by Default (RECOMMENDED)
**All galaxies are publicly viewable**, but users can mark individual nodes as private.
**Pros:**
- Simple implementation
- Encourages public knowledge sharing
- Better for SEO and discovery
- Aligns with "decentralized social" vision
**Cons:**
- Users might accidentally share private thoughts
- Requires clear UI indicators for node visibility
**URL Structure:**
- `/galaxy` - current user's galaxy (if logged in) or landing page
- `/galaxy/{user_did}` or `/galaxy?user={user_did}` - specific user's public galaxy
### Option 2: Opt-in Public Galleries
Galaxies are private by default, users must explicitly make them public.
**Pros:**
- More privacy-conscious
- Users have full control
**Cons:**
- Reduces discovery and sharing
- More complex implementation
- Goes against ATproto's "public by default" philosophy
**Decision: We'll implement Option 1** - Public by default, with optional private nodes.
## Implementation Plan
### Phase 1: Update API to Support Public Access
#### 1.1 Modify `/api/galaxy/route.ts`
```typescript
export async function GET(request: NextRequest) {
const { searchParams } = new URL(request.url);
const targetUserDid = searchParams.get('user');
const cookieStore = await cookies();
const surrealJwt = cookieStore.get('ponderants-auth')?.value;
// Determine which user's galaxy to show
let userDid: string;
let isOwnGalaxy = false;
if (targetUserDid) {
// Viewing someone else's public galaxy
userDid = targetUserDid;
} else if (surrealJwt) {
// Viewing own galaxy (authenticated)
const userSession = verifySurrealJwt(surrealJwt);
if (!userSession) {
return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
}
userDid = userSession.did;
isOwnGalaxy = true;
} else {
// No target user and not authenticated - return empty galaxy with message
return NextResponse.json({
nodes: [],
links: [],
message: 'Log in to view your galaxy, or visit a public galaxy via ?user={did}'
});
}
// Query nodes
const nodesQuery = `
SELECT id, title, body, user_did, atp_uri, coords_3d
FROM node
WHERE user_did = $userDid
AND coords_3d != NONE
${isOwnGalaxy ? '' : 'AND is_public = true'}
`;
// ... rest of implementation
}
```
#### 1.2 Add `is_public` Field to Node Schema
- Default: `true` (public by default)
- Users can mark individual nodes as private
- Private nodes are only visible to the owner
### Phase 2: Update Database Schema
#### 2.1 Add `is_public` Column to `node` Table
```sql
DEFINE FIELD is_public ON TABLE node TYPE bool DEFAULT true;
```
#### 2.2 Create Migration Script
`scripts/add-is-public-field.ts`:
```typescript
import { connectToDB } from '@/lib/db';
async function migrate() {
const db = await connectToDB();
// Add field definition
await db.query(`
DEFINE FIELD is_public ON TABLE node TYPE bool DEFAULT true;
`);
// Set existing nodes to public
await db.query(`
UPDATE node SET is_public = true WHERE is_public = NONE;
`);
console.log('Migration complete: Added is_public field');
}
migrate();
```
### Phase 3: Update Frontend Components
#### 3.1 Update `ThoughtGalaxy.tsx`
```typescript
// Support viewing other users' galaxies
const { searchParams } = useSearchParams();
const targetUser = searchParams.get('user');
useEffect(() => {
async function fetchData() {
const url = targetUser
? `/api/galaxy?user=${targetUser}`
: '/api/galaxy';
const response = await fetch(url, {
credentials: 'include',
});
// ... rest of implementation
}
fetchData();
}, [targetUser]);
```
#### 3.2 Add User Info Display
When viewing another user's galaxy, show:
- User's Bluesky handle
- Link to their profile
- Number of public nodes
#### 3.3 Update `/galaxy` Page
Add support for URL parameter: `/galaxy?user=did:plc:xxxxx`
### Phase 4: Navigation & User Experience
#### 4.1 Landing Experience for Non-Authenticated Users
When visiting `/galaxy` without login:
- Show a sample/demo galaxy (could be a curated example)
- Display call-to-action: "Create your own thought galaxy"
- Provide login button
#### 4.2 Add "Share Galaxy" Feature
Add button to copy shareable link:
```typescript
const shareUrl = `${window.location.origin}/galaxy?user=${userDid}`;
```
### Phase 5: Privacy Controls (Future Enhancement)
#### 5.1 Node-Level Privacy Toggle
In node editor, add checkbox:
```typescript
<Checkbox
label="Make this node public"
checked={isPublic}
onChange={(e) => setIsPublic(e.currentTarget.checked)}
/>
```
#### 5.2 Bulk Privacy Management
Settings page to:
- Make all nodes private/public
- Set default for new nodes
- Filter and update specific nodes
## Security Considerations
### 1. Data Exposure
- **Risk**: Users accidentally share sensitive information
- **Mitigation**:
- Clear visual indicators for public/private nodes
- Confirmation dialog when publishing nodes
- Easy way to make nodes private retroactively
### 2. API Abuse
- **Risk**: Scraping or excessive requests to public galaxies
- **Mitigation**:
- Rate limiting on `/api/galaxy`
- Caching layer for public galaxies
- Consider CDN for popular galaxies
### 3. Privacy Violations
- **Risk**: Viewing history tracking or surveillance
- **Mitigation**:
- No analytics on public galaxy views
- No "who viewed my galaxy" feature
- Respect DNT headers
## Testing Plan
### Magnitude Tests
#### Test 1: Public Galaxy Viewing Without Auth
```typescript
test('Unauthenticated users can view public galaxies', async (agent) => {
await agent.act('Navigate to /galaxy?user=did:plc:example123');
await agent.check('The galaxy visualization is displayed');
await agent.check('Public nodes are visible');
await agent.act('Click on a public node');
await agent.check('Node details are displayed');
});
```
#### Test 2: Private Nodes Hidden from Public View
```typescript
test('Private nodes are not visible in public galaxy', async (agent) => {
// ... implementation
});
```
#### Test 3: Own Galaxy Requires Auth
```typescript
test('Accessing own galaxy without target user requires authentication', async (agent) => {
await agent.act('Navigate to /galaxy');
await agent.check('Login prompt or empty state is displayed');
});
```
### Manual Testing Checklist
- [ ] Visit `/galaxy` without login → see landing page
- [ ] Visit `/galaxy?user={valid_did}` → see public nodes
- [ ] Visit `/galaxy?user={invalid_did}` → see error message
- [ ] Log in and visit `/galaxy` → see own galaxy (including private nodes)
- [ ] Share galaxy link → recipient can view public nodes
- [ ] Mark node as private → confirm it disappears from public view
## Implementation Steps
1. **Create database migration** for `is_public` field
2. **Update API route** to support public access
3. **Update ThoughtGalaxy component** to handle URL parameters
4. **Add user info display** for public galaxies
5. **Test with manual checks**
6. **Write Magnitude tests**
7. **Update documentation**
8. **Create PR with changes**
## Acceptance Criteria
✅ Unauthenticated users can view public galaxies via `?user=` parameter
✅ Authenticated users see their own galaxy at `/galaxy` (no param)
✅ Private nodes are only visible to the owner
✅ Public nodes are visible to everyone
✅ Clear error messages for invalid user DIDs
✅ Shareable URLs work correctly
✅ All tests pass
## Notes
- This aligns with ATproto's philosophy of public-by-default, user-controlled data
- Future enhancement: Node-level privacy controls in UI
- Consider adding Open Graph meta tags for social media previews
- May want to add a "featured galaxies" page for discovery
## Related Files
- `app/api/galaxy/route.ts` - Galaxy API endpoint
- `components/ThoughtGalaxy.tsx` - 3D visualization component
- `app/galaxy/page.tsx` - Galaxy page component
- `lib/db/schema.surql` - Database schema

View File

@@ -0,0 +1,44 @@
import Surreal from 'surrealdb';
/**
* Migration: Add is_public field to node table
*
* This script adds the is_public field to all existing nodes,
* defaulting them to public (true) to align with the public-by-default
* philosophy of ATproto.
*/
async function migrate() {
const db = new Surreal();
try {
await db.connect('ws://localhost:8000/rpc');
await db.signin({ username: 'root', password: 'root' });
await db.use({ namespace: 'ponderants', database: 'main' });
console.log('Adding is_public field definition to node table...');
// Define the field (should already be in schema.surql, but ensuring it's applied)
await db.query(`
DEFINE FIELD is_public ON TABLE node TYPE bool DEFAULT true;
`);
console.log('✓ Field definition added');
console.log('Setting existing nodes to public...');
// Set all existing nodes where is_public is NONE to true (public by default)
const result = await db.query(`
UPDATE node SET is_public = true WHERE is_public = NONE;
`);
console.log('Result:', result);
console.log('✓ Migration complete: All existing nodes are now public by default');
} catch (error) {
console.error('Migration failed:', error);
process.exit(1);
} finally {
await db.close();
}
}
migrate();

View File

@@ -0,0 +1,91 @@
import Surreal from 'surrealdb';
import { UMAP } from 'umap-js';
/**
* Recalculate 3D coordinates for ALL nodes
*
* This script fixes the issue where new nodes don't get coordinates
* because UMAP needs to see the full dataset to properly position points.
*
* Usage: tsx scripts/recalculate-all-coords.ts
*/
async function recalculateAllCoordinates() {
const db = new Surreal();
try {
// Connect to production database
const dbUrl = process.env.SURREALDB_URL || 'wss://ponderants-prod-06d6iecp19qj3bvmv2o0r5j50o.aws-usw2.surreal.cloud/rpc';
const dbNs = process.env.SURREALDB_NS || 'ponderants';
const dbName = process.env.SURREALDB_DB || 'production';
const dbUser = process.env.SURREALDB_USER || 'root';
const dbPass = process.env.SURREALDB_PASS;
if (!dbPass) {
throw new Error('SURREALDB_PASS environment variable is required');
}
console.log(`Connecting to ${dbUrl}...`);
await db.connect(dbUrl);
await db.signin({ username: dbUser, password: dbPass });
await db.use({ namespace: dbNs, database: dbName });
console.log('✓ Connected to database');
// Fetch ALL nodes with embeddings (not just those without coords)
console.log('Fetching all nodes with embeddings...');
const results = await db.query<[Array<{ id: string; embedding: number[] }>]>(
'SELECT id, title, embedding FROM node WHERE embedding != NONE'
);
const nodes = results[0] || [];
console.log(`Found ${nodes.length} nodes with embeddings`);
if (nodes.length === 0) {
console.log('No nodes with embeddings found');
return;
}
if (nodes.length < 3) {
console.error(`ERROR: Need at least 3 nodes for UMAP, found ${nodes.length}`);
return;
}
// Run UMAP on ALL nodes together
const embeddings = nodes.map((n) => n.embedding);
console.log('Running UMAP dimensionality reduction...');
console.log(`- Input: ${nodes.length} nodes with ${embeddings[0].length}-dimensional embeddings`);
console.log(`- Output: 3D coordinates`);
const umap = new UMAP({
nComponents: 3,
nNeighbors: Math.min(15, nodes.length - 1), // nNeighbors must be < sample size
minDist: 0.1,
spread: 1.0,
});
const coords_3d_array = await umap.fitAsync(embeddings);
console.log('✓ UMAP projection complete');
// Update ALL nodes with their new 3D coords
console.log('Updating nodes with new coordinates...');
for (let i = 0; i < nodes.length; i++) {
const node = nodes[i];
const coords = coords_3d_array[i];
await db.merge(node.id, {
coords_3d: [coords[0], coords[1], coords[2]],
});
console.log(` ✓ Updated ${node.id}: [${coords[0].toFixed(3)}, ${coords[1].toFixed(3)}, ${coords[2].toFixed(3)}]`);
}
console.log(`\n✅ Successfully updated ${nodes.length} nodes with 3D coordinates`);
} catch (error) {
console.error('❌ Error:', error);
process.exit(1);
} finally {
await db.close();
}
}
recalculateAllCoordinates();

View File

@@ -1,21 +1,126 @@
import { test } from 'magnitude-test'; import { test } from 'magnitude-test';
test('Mantine theme is applied correctly', async (agent) => { test('Theme toggle switches between light and dark modes', async (agent) => {
// Act: Navigate to the homepage // Act: Navigate to the homepage
await agent.act('Navigate to the homepage'); await agent.act('Navigate to the homepage');
// Check: Verify the Mantine components are rendered // Check: Verify the page loads with a theme
await agent.check('The text "Ponderants" is visible as a title'); await agent.check('The page has either a light or dark background');
await agent.check('A "Test Button" is visible on the screen');
// Check: Verify the theme is applied. // Act: Click the theme toggle button
// We check that the page uses a dark background with grayscale styling await agent.act('Click the theme toggle button');
await agent.check(
'The page has a dark background with light text, consistent with a grayscale dark theme'
);
// Check: Verify the Paper component is rendered with its themed styles // Wait for theme transition
await agent.check( await agent.act('Wait for 1 second');
'The page content is inside a "Paper" component with a border'
); // Check: Verify the theme has changed
await agent.check('The background color has changed to the opposite theme');
// Act: Click the theme toggle button again
await agent.act('Click the theme toggle button');
// Wait for theme transition
await agent.act('Wait for 1 second');
// Check: Verify the theme has changed back
await agent.check('The background color has changed back to the original theme');
});
test('Light mode displays correct colors', async (agent) => {
// Act: Navigate to the homepage
await agent.act('Navigate to the homepage');
// Act: Ensure we're in light mode by toggling if needed
await agent.act('If the background is dark, click the theme toggle button');
await agent.act('Wait for 1 second');
// Check: Verify light mode colors
await agent.check('The page has a light background color');
await agent.check('The sidebar has a light background');
await agent.check('The text is dark colored for readability');
await agent.check('The borders are subtle and light-colored');
});
test('Dark mode displays correct colors', async (agent) => {
// Act: Navigate to the homepage
await agent.act('Navigate to the homepage');
// Act: Ensure we're in dark mode by toggling if needed
await agent.act('If the background is light, click the theme toggle button');
await agent.act('Wait for 1 second');
// Check: Verify dark mode colors
await agent.check('The page has a dark background color');
await agent.check('The sidebar has a dark background');
await agent.check('The text is light colored for readability');
await agent.check('The borders are dark-colored');
});
test('Theme persists across page refreshes', async (agent) => {
// Act: Navigate to the homepage
await agent.act('Navigate to the homepage');
// Act: Set to light mode
await agent.act('If the background is dark, click the theme toggle button');
await agent.act('Wait for 1 second');
// Act: Refresh the page
await agent.act('Refresh the page');
await agent.act('Wait for the page to load');
// Check: Verify light mode persisted
await agent.check('The page still has a light background color');
// Act: Switch to dark mode
await agent.act('Click the theme toggle button');
await agent.act('Wait for 1 second');
// Act: Refresh the page again
await agent.act('Refresh the page');
await agent.act('Wait for the page to load');
// Check: Verify dark mode persisted
await agent.check('The page still has a dark background color');
});
test('Theme affects all UI components', async (agent) => {
// Act: Navigate to the homepage
await agent.act('Navigate to the homepage');
// Act: Ensure light mode
await agent.act('If the background is dark, click the theme toggle button');
await agent.act('Wait for 1 second');
// Check: Verify all components use light theme
await agent.check('The navigation sidebar uses light colors');
await agent.check('The main content area uses light colors');
await agent.check('All buttons and inputs use light theme styling');
// Act: Switch to dark mode
await agent.act('Click the theme toggle button');
await agent.act('Wait for 1 second');
// Check: Verify all components use dark theme
await agent.check('The navigation sidebar uses dark colors');
await agent.check('The main content area uses dark colors');
await agent.check('All buttons and inputs use dark theme styling');
});
test('Theme toggle icon changes based on current theme', async (agent) => {
// Act: Navigate to the homepage
await agent.act('Navigate to the homepage');
// Act: Ensure light mode
await agent.act('If the background is dark, click the theme toggle button');
await agent.act('Wait for 1 second');
// Check: Verify icon shows moon (indicating can switch to dark)
await agent.check('The theme toggle button shows a moon icon');
// Act: Switch to dark mode
await agent.act('Click the theme toggle button');
await agent.act('Wait for 1 second');
// Check: Verify icon shows sun (indicating can switch to light)
await agent.check('The theme toggle button shows a sun icon');
}); });

View File

@@ -0,0 +1,133 @@
import { test } from 'magnitude-test';
/**
* Public Galaxy Viewing Tests
*
* These tests verify that:
* 1. Galaxies can be viewed publicly via ?user={did} parameter
* 2. Private nodes are hidden from public view
* 3. Own galaxy requires authentication
* 4. Invalid user DIDs are handled gracefully
*/
test('Unauthenticated users can view public galaxies via user parameter', async (agent) => {
// Navigate to a public galaxy using the user parameter
// Note: This test assumes there's at least one user with public nodes
await agent.act('Navigate to /galaxy?user=did:plc:example123');
// Check: The galaxy visualization should be displayed
await agent.check('The 3D galaxy visualization is visible');
// Check: Public nodes should be visible
await agent.check('At least one node sphere is visible in the 3D space');
// Check: A "Public Galaxy" banner should be shown
await agent.check('A banner or indicator shows this is a public galaxy');
// Check: Can interact with nodes
await agent.act('Click on a visible node sphere');
await agent.check('A modal or panel displays the node title and content');
});
test('Private nodes are not visible in public galaxy view', async (agent) => {
// This test requires a user with both public and private nodes
// Navigate to their public galaxy
await agent.act('Navigate to /galaxy?user=did:plc:user-with-private-nodes');
// Check: Only public nodes are visible
await agent.check('The galaxy shows only public nodes');
// Check: Private nodes are not visible in the visualization
await agent.check('Private nodes are not rendered in the 3D space');
});
test('Accessing own galaxy without user parameter shows authenticated galaxy', async (agent) => {
// First, log in
await agent.act('Navigate to /');
await agent.act('Click the login button');
await agent.act('Complete the Bluesky OAuth flow');
// Navigate to /galaxy without user parameter
await agent.act('Navigate to /galaxy');
// Check: Should see own galaxy (including private nodes)
await agent.check('The galaxy visualization shows all nodes including private ones');
// Check: No "Public Galaxy" banner should be shown (it's your own galaxy)
await agent.check('There is no "Public Galaxy" banner visible');
});
test('Accessing own galaxy without authentication shows empty state', async (agent) => {
// Ensure user is logged out
await agent.act('Navigate to /');
await agent.act('If logged in, click logout');
// Navigate to /galaxy without user parameter
await agent.act('Navigate to /galaxy');
// Check: Should see a message about logging in or viewing a public galaxy
await agent.check('A message is displayed about logging in or visiting a public galaxy');
// Check: No nodes are visible
await agent.check('No node spheres are visible in the visualization');
});
test('Invalid user DID shows appropriate error or empty state', async (agent) => {
// Navigate to galaxy with an invalid/non-existent user DID
await agent.act('Navigate to /galaxy?user=did:plc:invalid-nonexistent-user');
// Check: Should handle gracefully (empty state or error message)
await agent.check('An appropriate message is shown for invalid or empty galaxy');
// Check: No crash or loading spinner stuck
await agent.check('The page is in a stable state (not stuck loading)');
});
test('Public galaxy displays user information', async (agent) => {
// Navigate to a public galaxy
await agent.act('Navigate to /galaxy?user=did:plc:example123');
// Check: User DID or handle is displayed
await agent.check('The user DID is visible in the UI');
// Check: Node count is displayed
await agent.check('The number of public nodes is shown');
});
test('Public galaxy node details show Bluesky link', async (agent) => {
// Navigate to a public galaxy
await agent.act('Navigate to /galaxy?user=did:plc:example123');
// Act: Click on a node
await agent.act('Click on a visible node sphere');
// Check: Node details modal is shown
await agent.check('A modal displays the node title and body');
// Check: "View on Bluesky" link is present
await agent.check('A "View on Bluesky" link is visible');
// Check: Link points to Bluesky
await agent.check('The link URL includes "bsky.app"');
});
test('Can switch between own galaxy and public galaxy', async (agent) => {
// Log in
await agent.act('Navigate to /');
await agent.act('Click the login button');
await agent.act('Complete the Bluesky OAuth flow');
// View own galaxy
await agent.act('Navigate to /galaxy');
await agent.check('Own galaxy is displayed');
// Navigate to someone else's public galaxy
await agent.act('Navigate to /galaxy?user=did:plc:other-user');
await agent.check('The "Public Galaxy" banner is shown');
await agent.check('A different set of nodes is visible');
// Navigate back to own galaxy
await agent.act('Navigate to /galaxy');
await agent.check('Own galaxy is displayed again');
await agent.check('No "Public Galaxy" banner is shown');
});