feat: Improve UI layout and navigation
- Increase logo size (48x48 desktop, 56x56 mobile) for better visibility - Add logo as favicon - Add logo to mobile header - Move user menu to navigation bars (sidebar on desktop, bottom bar on mobile) - Fix desktop chat layout - container structure prevents voice controls cutoff - Fix mobile bottom bar - use icon-only ActionIcons instead of truncated text buttons - Hide Create Node/New Conversation buttons on mobile to save header space - Make fixed header and voice controls work properly with containers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
349
docs/fixes/galaxy-graph-fix.md
Normal file
349
docs/fixes/galaxy-graph-fix.md
Normal file
@@ -0,0 +1,349 @@
|
||||
# Galaxy Graph Visualization Fix
|
||||
|
||||
## Problems
|
||||
|
||||
1. **Invalid URL Error**: ThoughtGalaxy component was failing with:
|
||||
```
|
||||
TypeError: Failed to construct 'URL': Invalid URL
|
||||
at parseUrl (surreal.ts:745:14)
|
||||
at Surreal.connectInner (surreal.ts:93:20)
|
||||
at Surreal.connect (surreal.ts:84:22)
|
||||
at fetchData (ThoughtGalaxy.tsx:76:16)
|
||||
```
|
||||
|
||||
2. **Manual Calculation Required**: Users had to manually click "Calculate My Graph" button to trigger UMAP dimensionality reduction
|
||||
|
||||
3. **"Not enough nodes" despite having 3+ nodes**: System was reporting insufficient nodes even after creating 3+ nodes with content
|
||||
|
||||
## Root Causes
|
||||
|
||||
### 1. Client-Side Database Connection
|
||||
The `ThoughtGalaxy.tsx` client component was attempting to connect directly to SurrealDB:
|
||||
|
||||
```typescript
|
||||
// ❌ Wrong: Client component trying to connect to database
|
||||
import Surreal from 'surrealdb';
|
||||
|
||||
useEffect(() => {
|
||||
const db = new Surreal();
|
||||
await db.connect(process.env.NEXT_PUBLIC_SURREALDB_WSS_URL!); // undefined!
|
||||
// ...
|
||||
}, []);
|
||||
```
|
||||
|
||||
Problems:
|
||||
- `NEXT_PUBLIC_SURREALDB_WSS_URL` environment variable didn't exist
|
||||
- Client components shouldn't connect directly to databases (security/architecture violation)
|
||||
- No authentication handling on client side
|
||||
|
||||
### 2. Manual Trigger Required
|
||||
Graph calculation only happened when user clicked a button. No automatic detection of when calculation was needed.
|
||||
|
||||
### 3. Connection Method Inconsistency
|
||||
The `calculate-graph` route was using inline database connection instead of the shared `connectToDB()` helper, leading to potential authentication mismatches.
|
||||
|
||||
## Solutions
|
||||
|
||||
### 1. Created Server-Side Galaxy API Route
|
||||
Created `/app/api/galaxy/route.ts` to handle all database access server-side:
|
||||
|
||||
```typescript
|
||||
export async function GET(request: NextRequest) {
|
||||
const cookieStore = await cookies();
|
||||
const surrealJwt = cookieStore.get('ponderants-auth')?.value;
|
||||
|
||||
if (!surrealJwt) {
|
||||
return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
|
||||
}
|
||||
|
||||
const userSession = verifySurrealJwt(surrealJwt);
|
||||
if (!userSession) {
|
||||
return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
|
||||
}
|
||||
|
||||
const { did: userDid } = userSession;
|
||||
|
||||
try {
|
||||
const db = await connectToDB();
|
||||
|
||||
// Fetch nodes that have 3D coordinates
|
||||
const nodesQuery = `
|
||||
SELECT id, title, coords_3d
|
||||
FROM node
|
||||
WHERE user_did = $userDid AND coords_3d != NONE
|
||||
`;
|
||||
const nodeResults = await db.query<[NodeData[]]>(nodesQuery, { userDid });
|
||||
const nodes = nodeResults[0] || [];
|
||||
|
||||
// Fetch links between nodes
|
||||
const linksQuery = `
|
||||
SELECT in, out
|
||||
FROM links_to
|
||||
`;
|
||||
const linkResults = await db.query<[LinkData[]]>(linksQuery);
|
||||
const links = linkResults[0] || [];
|
||||
|
||||
// Auto-trigger calculation if needed
|
||||
if (nodes.length === 0) {
|
||||
const unmappedQuery = `
|
||||
SELECT count() as count
|
||||
FROM node
|
||||
WHERE user_did = $userDid AND embedding != NONE AND coords_3d = NONE
|
||||
GROUP ALL
|
||||
`;
|
||||
const unmappedResults = await db.query<[Array<{ count: number }>]>(unmappedQuery, { userDid });
|
||||
const unmappedCount = unmappedResults[0]?.[0]?.count || 0;
|
||||
|
||||
if (unmappedCount >= 3) {
|
||||
console.log(`[Galaxy API] Found ${unmappedCount} unmapped nodes, triggering calculation...`);
|
||||
|
||||
// Trigger graph calculation (don't await, return current state)
|
||||
fetch(`${process.env.NEXT_PUBLIC_BASE_URL || 'http://localhost:3000'}/api/calculate-graph`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Cookie': `ponderants-auth=${surrealJwt}`,
|
||||
},
|
||||
}).catch((err) => {
|
||||
console.error('[Galaxy API] Failed to trigger graph calculation:', err);
|
||||
});
|
||||
|
||||
return NextResponse.json({
|
||||
nodes: [],
|
||||
links: [],
|
||||
message: 'Calculating 3D coordinates... Refresh in a moment.',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`[Galaxy API] Returning ${nodes.length} nodes and ${links.length} links`);
|
||||
|
||||
return NextResponse.json({
|
||||
nodes,
|
||||
links,
|
||||
});
|
||||
} catch (error) {
|
||||
console.error('[Galaxy API] Error:', error);
|
||||
return NextResponse.json(
|
||||
{ error: 'Failed to fetch galaxy data' },
|
||||
{ status: 500 }
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Key features:
|
||||
- ✅ Server-side authentication with JWT verification
|
||||
- ✅ Data isolation via `user_did` filtering
|
||||
- ✅ Auto-detection of unmapped nodes
|
||||
- ✅ Automatic triggering of UMAP calculation
|
||||
- ✅ Progress messaging for client polling
|
||||
|
||||
### 2. Updated ThoughtGalaxy Component
|
||||
Changed from direct database connection to API-based data fetching:
|
||||
|
||||
**Before:**
|
||||
```typescript
|
||||
import Surreal from 'surrealdb';
|
||||
|
||||
useEffect(() => {
|
||||
async function fetchData() {
|
||||
const db = new Surreal();
|
||||
await db.connect(process.env.NEXT_PUBLIC_SURREALDB_WSS_URL!);
|
||||
const token = document.cookie.split('ponderants-auth=')[1];
|
||||
await db.authenticate(token);
|
||||
|
||||
const nodeResults = await db.query('SELECT id, title, coords_3d FROM node...');
|
||||
setNodes(nodeResults[0] || []);
|
||||
}
|
||||
fetchData();
|
||||
}, []);
|
||||
```
|
||||
|
||||
**After:**
|
||||
```typescript
|
||||
// No Surreal import needed
|
||||
|
||||
useEffect(() => {
|
||||
async function fetchData() {
|
||||
try {
|
||||
const response = await fetch('/api/galaxy', {
|
||||
credentials: 'include', // Include cookies for authentication
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
console.error('[ThoughtGalaxy] Failed to fetch galaxy data:', response.statusText);
|
||||
return;
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (data.message) {
|
||||
console.log('[ThoughtGalaxy]', data.message);
|
||||
// If calculating, poll again in 2 seconds
|
||||
setTimeout(fetchData, 2000);
|
||||
return;
|
||||
}
|
||||
|
||||
setNodes(data.nodes || []);
|
||||
setLinks(data.links || []);
|
||||
|
||||
console.log(`[ThoughtGalaxy] Loaded ${data.nodes?.length || 0} nodes and ${data.links?.length || 0} links`);
|
||||
} catch (error) {
|
||||
console.error('[ThoughtGalaxy] Error fetching data:', error);
|
||||
}
|
||||
}
|
||||
|
||||
fetchData();
|
||||
}, []);
|
||||
```
|
||||
|
||||
Key improvements:
|
||||
- ✅ No client-side database connection
|
||||
- ✅ Proper authentication via HTTP-only cookies
|
||||
- ✅ Polling mechanism for in-progress calculations
|
||||
- ✅ Better error handling
|
||||
|
||||
### 3. Fixed calculate-graph Route
|
||||
Updated `/app/api/calculate-graph/route.ts` to use shared helpers:
|
||||
|
||||
**Before:**
|
||||
```typescript
|
||||
const db = new (await import('surrealdb')).default();
|
||||
await db.connect(process.env.SURREALDB_URL!);
|
||||
await db.signin({
|
||||
username: process.env.SURREALDB_USER!,
|
||||
password: process.env.SURREALDB_PASS!,
|
||||
});
|
||||
|
||||
const jwt = require('jsonwebtoken');
|
||||
const decoded = jwt.decode(surrealJwt) as { did: string };
|
||||
const userDid = decoded?.did;
|
||||
```
|
||||
|
||||
**After:**
|
||||
```typescript
|
||||
import { connectToDB } from '@/lib/db';
|
||||
import { verifySurrealJwt } from '@/lib/auth/jwt';
|
||||
|
||||
const userSession = verifySurrealJwt(surrealJwt);
|
||||
if (!userSession) {
|
||||
return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
|
||||
}
|
||||
|
||||
const { did: userDid } = userSession;
|
||||
|
||||
const db = await connectToDB();
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- ✅ Consistent authentication across all routes
|
||||
- ✅ Proper JWT verification (not just decode)
|
||||
- ✅ Reusable code (DRY principle)
|
||||
|
||||
### 4. Created Debug Endpoint
|
||||
Added `/app/api/debug/nodes/route.ts` for database inspection:
|
||||
|
||||
```typescript
|
||||
export async function GET(request: NextRequest) {
|
||||
const cookieStore = await cookies();
|
||||
const surrealJwt = cookieStore.get('ponderants-auth')?.value;
|
||||
|
||||
if (!surrealJwt) {
|
||||
return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
|
||||
}
|
||||
|
||||
const userSession = verifySurrealJwt(surrealJwt);
|
||||
if (!userSession) {
|
||||
return NextResponse.json({ error: 'Invalid auth token' }, { status: 401 });
|
||||
}
|
||||
|
||||
const { did: userDid } = userSession;
|
||||
|
||||
try {
|
||||
const db = await connectToDB();
|
||||
|
||||
const nodesQuery = `
|
||||
SELECT id, title, body, atp_uri, embedding, coords_3d
|
||||
FROM node
|
||||
WHERE user_did = $userDid
|
||||
`;
|
||||
const results = await db.query(nodesQuery, { userDid });
|
||||
const nodes = results[0] || [];
|
||||
|
||||
const stats = {
|
||||
total: nodes.length,
|
||||
with_embeddings: nodes.filter((n: any) => n.embedding).length,
|
||||
with_coords: nodes.filter((n: any) => n.coords_3d).length,
|
||||
without_embeddings: nodes.filter((n: any) => !n.embedding).length,
|
||||
without_coords: nodes.filter((n: any) => !n.coords_3d).length,
|
||||
};
|
||||
|
||||
return NextResponse.json({
|
||||
stats,
|
||||
nodes: nodes.map((n: any) => ({
|
||||
id: n.id,
|
||||
title: n.title,
|
||||
atp_uri: n.atp_uri,
|
||||
has_embedding: !!n.embedding,
|
||||
has_coords: !!n.coords_3d,
|
||||
coords_3d: n.coords_3d,
|
||||
})),
|
||||
});
|
||||
} catch (error) {
|
||||
console.error('[Debug Nodes] Error:', error);
|
||||
return NextResponse.json({ error: String(error) }, { status: 500 });
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use: Visit `/api/debug/nodes` while logged in to see your node statistics and data.
|
||||
|
||||
## Auto-Calculation Flow
|
||||
|
||||
1. **User visits Galaxy page** → ThoughtGalaxy component mounts
|
||||
2. **Component fetches data** → `GET /api/galaxy`
|
||||
3. **API checks for coords** → Query: `WHERE coords_3d != NONE`
|
||||
4. **If no coords found** → Query unmapped count: `WHERE embedding != NONE AND coords_3d = NONE`
|
||||
5. **If ≥3 unmapped nodes** → Trigger `POST /api/calculate-graph` (don't wait)
|
||||
6. **Return progress message** → `{ message: 'Calculating 3D coordinates...' }`
|
||||
7. **Client polls** → setTimeout 2 seconds, fetch again
|
||||
8. **UMAP completes** → Next poll returns actual node data
|
||||
9. **Client renders** → 3D visualization appears
|
||||
|
||||
## Files Changed
|
||||
|
||||
1. `/components/ThoughtGalaxy.tsx` - Removed direct DB connection, added API-based fetching and polling
|
||||
2. `/app/api/galaxy/route.ts` - **NEW** - Server-side galaxy data endpoint with auto-calculation
|
||||
3. `/app/api/calculate-graph/route.ts` - Updated to use `connectToDB()` and `verifySurrealJwt()`
|
||||
4. `/app/api/debug/nodes/route.ts` - **NEW** - Debug endpoint for inspecting node data
|
||||
|
||||
## Verification
|
||||
|
||||
After the fix:
|
||||
|
||||
```bash
|
||||
# Server logs show auto-calculation:
|
||||
[Galaxy API] Found 5 unmapped nodes, triggering calculation...
|
||||
[Calculate Graph] Processing 5 nodes for UMAP projection
|
||||
[Calculate Graph] Running UMAP dimensionality reduction...
|
||||
[Calculate Graph] ✓ UMAP projection complete
|
||||
[Calculate Graph] ✓ Updated 5 nodes with 3D coordinates
|
||||
```
|
||||
|
||||
```bash
|
||||
# Client logs show polling:
|
||||
[ThoughtGalaxy] Calculating 3D coordinates... Refresh in a moment.
|
||||
[ThoughtGalaxy] Loaded 5 nodes and 3 links
|
||||
```
|
||||
|
||||
## Architecture Note
|
||||
|
||||
This fix maintains the "Source of Truth vs. App View Cache" pattern:
|
||||
- **ATproto PDS** - Canonical source of Node content (com.ponderants.node records)
|
||||
- **SurrealDB** - Performance cache that stores:
|
||||
- Copy of node data for fast access
|
||||
- Vector embeddings for similarity search
|
||||
- Pre-computed 3D coordinates for visualization
|
||||
- Graph links between nodes
|
||||
|
||||
The auto-calculation ensures the cache stays enriched with visualization data without user intervention.
|
||||
122
docs/fixes/surrealdb-cache-fix.md
Normal file
122
docs/fixes/surrealdb-cache-fix.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# SurrealDB Cache Authentication Fix
|
||||
|
||||
## Problem
|
||||
|
||||
Node publishing was showing a yellow warning notification:
|
||||
```
|
||||
Node published to Bluesky, but cache update failed. Advanced features may be unavailable.
|
||||
```
|
||||
|
||||
Server logs showed:
|
||||
```
|
||||
[POST /api/nodes] ⚠ SurrealDB cache write failed (non-critical):
|
||||
Error [ResponseError]: There was a problem with the database: There was a problem with authentication
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
The `connectToDB()` function in `lib/db.ts` was attempting to authenticate with SurrealDB using our application's custom JWT token:
|
||||
|
||||
```typescript
|
||||
await db.authenticate(token);
|
||||
```
|
||||
|
||||
However, SurrealDB doesn't know how to validate our custom JWT tokens. The `db.authenticate()` method is for SurrealDB's own token-based authentication system, not for validating external JWTs.
|
||||
|
||||
## Solution
|
||||
|
||||
Changed `connectToDB()` to use root credentials instead:
|
||||
|
||||
### Before:
|
||||
```typescript
|
||||
export async function connectToDB(token: string): Promise<Surreal> {
|
||||
const db = new Surreal();
|
||||
await db.connect(SURREALDB_URL);
|
||||
await db.authenticate(token); // ❌ This fails
|
||||
await db.use({ namespace, database });
|
||||
return db;
|
||||
}
|
||||
```
|
||||
|
||||
### After:
|
||||
```typescript
|
||||
export async function connectToDB(): Promise<Surreal> {
|
||||
const db = new Surreal();
|
||||
await db.connect(SURREALDB_URL);
|
||||
await db.signin({
|
||||
username: SURREALDB_USER, // ✅ Use root credentials
|
||||
password: SURREALDB_PASS,
|
||||
});
|
||||
await db.use({ namespace, database });
|
||||
return db;
|
||||
}
|
||||
```
|
||||
|
||||
## Data Security
|
||||
|
||||
Since we're now using root credentials instead of JWT-based authentication, we maintain data isolation by:
|
||||
|
||||
1. **Extracting user_did from the verified JWT** in API routes
|
||||
2. **Filtering all queries by user_did** to ensure users only access their own data
|
||||
|
||||
Example from `/app/api/nodes/route.ts`:
|
||||
```typescript
|
||||
// Verify JWT to get user's DID
|
||||
const userSession = verifySurrealJwt(surrealJwt);
|
||||
const { did: userDid } = userSession;
|
||||
|
||||
// Create node with user_did field
|
||||
const nodeData = {
|
||||
user_did: userDid, // ✅ Enforces data ownership
|
||||
atp_uri: atp_uri,
|
||||
title: title,
|
||||
body: body,
|
||||
embedding: embedding,
|
||||
};
|
||||
await db.create('node', nodeData);
|
||||
```
|
||||
|
||||
Example from `/app/api/suggest-links/route.ts`:
|
||||
```typescript
|
||||
// Query filtered by user_did
|
||||
const query = `
|
||||
SELECT * FROM node
|
||||
WHERE user_did = $user_did // ✅ Only user's own nodes
|
||||
ORDER BY score DESC
|
||||
LIMIT 5;
|
||||
`;
|
||||
await db.query(query, { user_did: userDid });
|
||||
```
|
||||
|
||||
## Files Changed
|
||||
|
||||
1. `/lib/db.ts` - Changed authentication method
|
||||
2. `/app/api/nodes/route.ts` - Removed JWT parameter from `connectToDB()` call
|
||||
3. `/app/api/suggest-links/route.ts` - Updated to use root credentials and filter by `user_did`
|
||||
|
||||
## Test
|
||||
|
||||
Created `/tests/magnitude/cache-success.mag.ts` to verify:
|
||||
- Node publishes successfully
|
||||
- GREEN success notification (not yellow warning)
|
||||
- No "cache update failed" message
|
||||
|
||||
## Verification
|
||||
|
||||
After the fix, server logs show:
|
||||
```
|
||||
[POST /api/nodes] ✓ Published to ATproto PDS
|
||||
[POST /api/nodes] ✓ Generated embedding vector
|
||||
[POST /api/nodes] ✓ Cached node in SurrealDB
|
||||
POST /api/nodes 200 in 1078ms
|
||||
```
|
||||
|
||||
No more authentication errors! 🎉
|
||||
|
||||
## Architecture Note
|
||||
|
||||
This implements the "App View Cache" pattern where:
|
||||
- **ATproto PDS** is the source of truth (decentralized, user-owned)
|
||||
- **SurrealDB** is a performance cache (centralized, app-managed)
|
||||
|
||||
The cache uses root credentials but enforces data isolation through `user_did` filtering in application code, similar to how the OAuth session store works (`lib/auth/oauth-session-store.ts`).
|
||||
@@ -1,208 +1,33 @@
|
||||
# **File: COMMIT\_10\_LINKING.md**
|
||||
# **File: COMMIT\_11\_VIZ.md**
|
||||
|
||||
## **Commit 10: Node Editor & AI-Powered Linking**
|
||||
## **Commit 11: 3D "Thought Galaxy" Visualization**
|
||||
|
||||
### **Objective**
|
||||
|
||||
Build the node editor UI and the AI-powered "Find related" feature. This commit will:
|
||||
Implement the 3D "Thought Galaxy" visualization using React Three Fiber (R3F). This commit addresses **Risk 3 (UMAP Projection)** by using the "Calculate My Graph" button strategy for the hackathon.
|
||||
|
||||
1. Create the editor page (/editor/\[id\]) that is pre-filled by the chat (Commit 07\) or loaded from the DB.
|
||||
2. Implement the "Publish" button, which calls the /api/nodes route (from Commit 06).
|
||||
3. Implement the "Find related" button, which calls a *new* /api/suggest-links route.
|
||||
4. Implement the /api/suggest-links route, which generates an embedding for the current draft and uses SurrealDB's vector search to find similar nodes.15
|
||||
1. Create an API route /api/calculate-graph that:
|
||||
* Fetches all the user's node embeddings from SurrealDB.
|
||||
* Uses umap-js to run dimensionality reduction from 1536-D down to 3-D.26
|
||||
* Updates the coords\_3d field for each node in SurrealDB.
|
||||
2. Create a client-side R3F component (/app/galaxy) that:
|
||||
* Fetches all nodes *with* coords\_3d coordinates.
|
||||
* Renders each node as a \<mesh\>.
|
||||
* Renders links as \<Line\>.
|
||||
* Uses \<CameraControls\> for smooth onClick interaction.
|
||||
|
||||
### **Implementation Specification**
|
||||
|
||||
**1\. Create Editor Page (app/editor/\[id\]/page.tsx)**
|
||||
**1\. Create Graph Calculation API (app/api/calculate-graph/route.ts)**
|
||||
|
||||
Create a file at /app/editor/\[id\]/page.tsx:
|
||||
|
||||
TypeScript
|
||||
|
||||
'use client';
|
||||
|
||||
import {
|
||||
Container,
|
||||
Title,
|
||||
TextInput,
|
||||
Textarea,
|
||||
Button,
|
||||
Stack,
|
||||
Paper,
|
||||
Text,
|
||||
LoadingOverlay,
|
||||
Group,
|
||||
} from '@mantine/core';
|
||||
import { useForm } from '@mantine/form';
|
||||
import { useSearchParams, useRouter, useParams } from 'next/navigation';
|
||||
import { useEffect, useState } from 'react';
|
||||
|
||||
// Define the shape of a suggested link
|
||||
interface SuggestedNode {
|
||||
id: string;
|
||||
title: string;
|
||||
body: string;
|
||||
score: number;
|
||||
}
|
||||
|
||||
export default function EditorPage() {
|
||||
const router \= useRouter();
|
||||
const params \= useParams();
|
||||
const searchParams \= useSearchParams();
|
||||
|
||||
const \[isPublishing, setIsPublishing\] \= useState(false);
|
||||
const \[isFinding, setIsFinding\] \= useState(false);
|
||||
const \= useState\<SuggestedNode\>();
|
||||
|
||||
const form \= useForm({
|
||||
initialValues: {
|
||||
title: '',
|
||||
body: '',
|
||||
links: as string, // Array of at-uri strings
|
||||
},
|
||||
});
|
||||
|
||||
// Pre-fill form from search params (from AI chat redirect)
|
||||
useEffect(() \=\> {
|
||||
if (params.id \=== 'new') {
|
||||
const title \= searchParams.get('title') |
|
||||
|
||||
| '';
|
||||
const body \= searchParams.get('body') |
|
||||
|
||||
| '';
|
||||
form.setValues({ title, body });
|
||||
} else {
|
||||
// TODO: Load existing node from /api/nodes/\[id\]
|
||||
}
|
||||
}, \[params.id, searchParams\]);
|
||||
|
||||
// Handler for the "Publish" button (calls Commit 06 API)
|
||||
const handlePublish \= async (values: typeof form.values) \=\> {
|
||||
setIsPublishing(true);
|
||||
try {
|
||||
const response \= await fetch('/api/nodes', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(values),
|
||||
});
|
||||
|
||||
if (\!response.ok) {
|
||||
throw new Error('Failed to publish node');
|
||||
}
|
||||
|
||||
const newNode \= await response.json();
|
||||
// On success, go to the graph
|
||||
router.push('/galaxy');
|
||||
|
||||
} catch (error) {
|
||||
console.error(error);
|
||||
// TODO: Show notification
|
||||
} finally {
|
||||
setIsPublishing(false);
|
||||
}
|
||||
};
|
||||
|
||||
// Handler for the "Find related" button
|
||||
const handleFindRelated \= async () \=\> {
|
||||
setIsFinding(true);
|
||||
setSuggestions();
|
||||
try {
|
||||
const response \= await fetch('/api/suggest-links', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ body: form.values.body }),
|
||||
});
|
||||
|
||||
if (\!response.ok) {
|
||||
throw new Error('Failed to find links');
|
||||
}
|
||||
|
||||
const relatedNodes \= await response.json();
|
||||
setSuggestions(relatedNodes);
|
||||
|
||||
} catch (error) {
|
||||
console.error(error);
|
||||
// TODO: Show notification
|
||||
} finally {
|
||||
setIsFinding(false);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
\<Container size="md" py="xl"\>
|
||||
\<form onSubmit={form.onSubmit(handlePublish)}\>
|
||||
\<Stack gap="md"\>
|
||||
\<Title order={2}\>
|
||||
{params.id \=== 'new'? 'Create New Node' : 'Edit Node'}
|
||||
\</Title\>
|
||||
|
||||
\<TextInput
|
||||
label="Title"
|
||||
placeholder="Your node title"
|
||||
required
|
||||
{...form.getInputProps('title')}
|
||||
/\>
|
||||
|
||||
\<Textarea
|
||||
label="Body"
|
||||
placeholder="Your node content..."
|
||||
required
|
||||
minRows={10}
|
||||
autosize
|
||||
{...form.getInputProps('body')}
|
||||
/\>
|
||||
|
||||
\<Group\>
|
||||
\<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
onClick={handleFindRelated}
|
||||
loading={isFinding}
|
||||
\>
|
||||
Find Related
|
||||
\</Button\>
|
||||
\<Button type="submit" loading={isPublishing}\>
|
||||
Publish Node
|
||||
\</Button\>
|
||||
\</Group\>
|
||||
|
||||
{/\* Related Links Section \*/}
|
||||
\<Stack\>
|
||||
{isFinding && \<LoadingOverlay visible /\>}
|
||||
{suggestions.length \> 0 && \<Title order={4}\>Suggested Links\</Title\>}
|
||||
|
||||
{suggestions.map((node) \=\> (
|
||||
\<Paper key={node.id} withBorder p="sm"\>
|
||||
\<Text fw={700}\>{node.title}\</Text\>
|
||||
\<Text size="sm" lineClamp={2}\>{node.body}\</Text\>
|
||||
\<Text size="xs" c="dimmed"\>Similarity: {(node.score \* 100).toFixed(0)}%\</Text\>
|
||||
\</Paper\>
|
||||
))}
|
||||
|
||||
{\!isFinding && suggestions.length \=== 0 && (
|
||||
\<Text size="sm" c="dimmed"\>
|
||||
{/\* Placeholder text \*/}
|
||||
\</Text\>
|
||||
)}
|
||||
\</Stack\>
|
||||
|
||||
\</Stack\>
|
||||
\</form\>
|
||||
\</Container\>
|
||||
);
|
||||
}
|
||||
|
||||
**2\. Create Link Suggestion API (app/api/suggest-links/route.ts)**
|
||||
|
||||
Create a file at /app/api/suggest-links/route.ts:
|
||||
Create a file at /app/api/calculate-graph/route.ts:
|
||||
|
||||
TypeScript
|
||||
|
||||
import { NextRequest, NextResponse } from 'next/server';
|
||||
import { cookies } from 'next/headers';
|
||||
import { connectToDB } from '@/lib/db';
|
||||
import { generateEmbedding } from '@/lib/ai';
|
||||
import { UMAP } from 'umap-js';
|
||||
|
||||
export async function POST(request: NextRequest) {
|
||||
const surrealJwt \= cookies().get('ponderants-auth')?.value;
|
||||
@@ -210,137 +35,329 @@ export async function POST(request: NextRequest) {
|
||||
return NextResponse.json({ error: 'Not authenticated' }, { status: 401 });
|
||||
}
|
||||
|
||||
const { body } \= await request.json();
|
||||
if (\!body) {
|
||||
return NextResponse.json({ error: 'Body text is required' }, { status: 400 });
|
||||
}
|
||||
|
||||
try {
|
||||
// 1\. Generate embedding for the current draft
|
||||
const draftEmbedding \= await generateEmbedding(body);
|
||||
|
||||
// 2\. Connect to DB (as the user)
|
||||
const db \= await connectToDB(surrealJwt);
|
||||
|
||||
// 3\. Run the vector similarity search query
|
||||
// This query finds the 5 closest nodes in the 'node' table
|
||||
// using cosine similarity on the 'embedding' field.
|
||||
// It only searches nodes WHERE user\_did \= $token.did,
|
||||
// which is enforced by the table's PERMISSIONS.
|
||||
const query \= \`
|
||||
SELECT
|
||||
id,
|
||||
title,
|
||||
body,
|
||||
atp\_uri,
|
||||
vector::similarity::cosine(embedding, $draft\_embedding) AS score
|
||||
FROM node
|
||||
ORDER BY score DESC
|
||||
LIMIT 5;
|
||||
\`;
|
||||
// 1\. Fetch all nodes that have an embedding but no coords
|
||||
const query \= \`SELECT id, embedding FROM node WHERE embedding\!= NONE AND coords\_3d \= NONE\`;
|
||||
const results \= await db.query(query);
|
||||
|
||||
const results \= await db.query(query, {
|
||||
draft\_embedding: draftEmbedding,
|
||||
const nodes \= results.result as { id: string; embedding: number };
|
||||
|
||||
if (nodes.length \< 3) {
|
||||
// UMAP needs a few points to work well
|
||||
return NextResponse.json({ message: 'Not enough nodes to map.' });
|
||||
}
|
||||
|
||||
// 2\. Prepare data for UMAP
|
||||
const embeddings \= nodes.map(n \=\> n.embedding);
|
||||
|
||||
// 3\. Run UMAP to reduce 1536-D to 3-D \[26\]
|
||||
const umap \= new UMAP({
|
||||
nComponents: 3,
|
||||
nNeighbors: Math.min(15, nodes.length \- 1), // nNeighbors must be \< sample size
|
||||
minDist: 0.1,
|
||||
spread: 1.0,
|
||||
});
|
||||
|
||||
// The query returns an array of result sets. We want the first one.
|
||||
return NextResponse.json(results.result);
|
||||
const coords\_3d\_array \= umap.fit(embeddings);
|
||||
|
||||
// 4\. Update nodes in SurrealDB with their new 3D coords
|
||||
// This is run in a transaction for speed.
|
||||
let transaction \= 'BEGIN TRANSACTION;';
|
||||
|
||||
nodes.forEach((node, index) \=\> {
|
||||
const coords \= coords\_3d\_array\[index\];
|
||||
transaction \+= \`UPDATE ${node.id} SET coords\_3d \= \[${coords}, ${coords}, ${coords}\];\`;
|
||||
});
|
||||
|
||||
transaction \+= 'COMMIT TRANSACTION;';
|
||||
|
||||
await db.query(transaction);
|
||||
|
||||
return NextResponse.json({ success: true, nodes\_mapped: nodes.length });
|
||||
|
||||
} catch (error) {
|
||||
console.error('Link suggestion error:', error);
|
||||
console.error('Graph calculation error:', error);
|
||||
return NextResponse.json(
|
||||
{ error: 'Failed to suggest links' },
|
||||
{ error: 'Failed to calculate graph' },
|
||||
{ status: 500 }
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
**2\. Create Galaxy Page (app/galaxy/page.tsx)**
|
||||
|
||||
Create a file at /app/galaxy/page.tsx:
|
||||
|
||||
TypeScript
|
||||
|
||||
'use client';
|
||||
|
||||
import { Button, Box } from '@mantine/core';
|
||||
import { Suspense, useState } from 'react';
|
||||
import { ThoughtGalaxy } from '@/components/ThoughtGalaxy';
|
||||
|
||||
export default function GalaxyPage() {
|
||||
const \[isCalculating, setIsCalculating\] \= useState(false);
|
||||
// This key forces a re-render of the galaxy component
|
||||
const \[galaxyKey, setGalaxyKey\] \= useState(Date.now());
|
||||
|
||||
const handleCalculateGraph \= async () \=\> {
|
||||
setIsCalculating(true);
|
||||
try {
|
||||
await fetch('/api/calculate-graph', { method: 'POST' });
|
||||
// Refresh the galaxy component by changing its key
|
||||
setGalaxyKey(Date.now());
|
||||
} catch (error) {
|
||||
console.error(error);
|
||||
// TODO: Show notification
|
||||
} finally {
|
||||
setIsCalculating(false);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
\<Box style={{ height: '100vh', width: '100vw', position: 'relative' }}\>
|
||||
\<Button
|
||||
onClick={handleCalculateGraph}
|
||||
loading={isCalculating}
|
||||
style={{ position: 'absolute', top: 20, left: 20, zIndex: 10 }}
|
||||
\>
|
||||
Calculate My Graph
|
||||
\</Button\>
|
||||
|
||||
{/\* R3F Canvas for the 3D visualization \*/}
|
||||
\<Suspense fallback={\<Box\>Loading 3D Scene...\</Box\>}\>
|
||||
\<ThoughtGalaxy key={galaxyKey} /\>
|
||||
\</SuspE\>
|
||||
\</Box\>
|
||||
);
|
||||
}
|
||||
|
||||
**3\. Create 3D Component (components/ThoughtGalaxy.tsx)**
|
||||
|
||||
Create a file at /components/ThoughtGalaxy.tsx:
|
||||
|
||||
TypeScript
|
||||
|
||||
'use client';
|
||||
|
||||
import { Canvas, useLoader } from '@react-three/fiber';
|
||||
import {
|
||||
CameraControls,
|
||||
Line,
|
||||
Text,
|
||||
useCursor,
|
||||
} from '@react-three/drei';
|
||||
import { Suspense, useEffect, useRef, useState } from 'react';
|
||||
import \* as THREE from 'three';
|
||||
import { Surreal } from 'surrealdb.js';
|
||||
|
||||
// Define the shape of nodes and links from DB
|
||||
interface NodeData {
|
||||
id: string;
|
||||
title: string;
|
||||
coords\_3d: \[number, number, number\];
|
||||
}
|
||||
interface LinkData {
|
||||
in: string; // from node id
|
||||
out: string; // to node id
|
||||
}
|
||||
|
||||
// 1\. The 3D Node Component
|
||||
function Node({ node, onNodeClick }) {
|
||||
const \[hovered, setHovered\] \= useState(false);
|
||||
const \[clicked, setClicked\] \= useState(false);
|
||||
useCursor(hovered);
|
||||
|
||||
return (
|
||||
\<mesh
|
||||
position={node.coords\_3d}
|
||||
onClick={(e) \=\> {
|
||||
e.stopPropagation();
|
||||
onNodeClick(node);
|
||||
setClicked(\!clicked);
|
||||
}}
|
||||
onPointerOver={(e) \=\> {
|
||||
e.stopPropagation();
|
||||
setHovered(true);
|
||||
}}
|
||||
onPointerOut={() \=\> setHovered(false)}
|
||||
\>
|
||||
\<sphereGeometry args={\[0.1, 32, 32\]} /\>
|
||||
\<meshStandardMaterial
|
||||
color={hovered? '\#90c0ff' : '\#e9ecef'}
|
||||
emissive={hovered? '\#90c0ff' : '\#e9ecef'}
|
||||
emissiveIntensity={hovered? 0.5 : 0.1}
|
||||
/\>
|
||||
{/\* Show title on hover or click \*/}
|
||||
{(hovered |
|
||||
|
||||
| clicked) && (
|
||||
\<Text
|
||||
position={\[0, 0.2, 0\]}
|
||||
fontSize={0.1}
|
||||
color="white"
|
||||
anchorX="center"
|
||||
anchorY="middle"
|
||||
\>
|
||||
{node.title}
|
||||
\</Text\>
|
||||
)}
|
||||
\</mesh\>
|
||||
);
|
||||
}
|
||||
|
||||
// 2\. The Main Scene Component
|
||||
export function ThoughtGalaxy() {
|
||||
const \[nodes, setNodes\] \= useState\<NodeData\>();
|
||||
const \[links, setLinks\] \= useState\<LinkData\>();
|
||||
const cameraControlsRef \= useRef\<CameraControls\>(null);
|
||||
|
||||
// Fetch data from SurrealDB on mount
|
||||
useEffect(() \=\> {
|
||||
async function fetchData() {
|
||||
// Client-side connection
|
||||
const db \= new Surreal();
|
||||
await db.connect(process.env.NEXT\_PUBLIC\_SURREALDB\_WSS\_URL\!);
|
||||
|
||||
// Get the token from the cookie (this is a hack,
|
||||
// proper way is to use an API route)
|
||||
const token \= document.cookie
|
||||
.split('; ')
|
||||
.find(row \=\> row.startsWith('ponderants-auth='))
|
||||
?.split('=');
|
||||
|
||||
if (\!token) return;
|
||||
|
||||
await db.authenticate(token);
|
||||
|
||||
// Fetch nodes that have coordinates
|
||||
const nodeResults \= await db.query(
|
||||
'SELECT id, title, coords\_3d FROM node WHERE coords\_3d\!= NONE'
|
||||
);
|
||||
setNodes((nodeResults.result as NodeData) ||);
|
||||
|
||||
// Fetch links
|
||||
const linkResults \= await db.query('SELECT in, out FROM links\_to');
|
||||
setLinks((linkResults.result as LinkData) ||);
|
||||
}
|
||||
fetchData();
|
||||
},);
|
||||
|
||||
// Map links to node positions
|
||||
const linkLines \= links
|
||||
.map((link) \=\> {
|
||||
const startNode \= nodes.find((n) \=\> n.id \=== link.in);
|
||||
const endNode \= nodes.find((n) \=\> n.id \=== link.out);
|
||||
if (startNode && endNode) {
|
||||
return {
|
||||
start: startNode.coords\_3d,
|
||||
end: endNode.coords\_3d,
|
||||
};
|
||||
}
|
||||
return null;
|
||||
})
|
||||
.filter(Boolean) as { start: \[number, number, number\]; end: \[number, number, number\] };
|
||||
|
||||
// Camera animation
|
||||
const handleNodeClick \= (node: NodeData) \=\> {
|
||||
cameraControlsRef.current?.smoothTime \= 0.8;
|
||||
cameraControlsRef.current?.setLookAt(
|
||||
node.coords\_3d \+ 1,
|
||||
node.coords\_3d \+ 1,
|
||||
node.coords\_3d \+ 1,
|
||||
...node.coords\_3d,
|
||||
true // Enable smooth transition
|
||||
);
|
||||
};
|
||||
|
||||
return (
|
||||
\<Canvas camera={{ position: , fov: 60 }}\>
|
||||
\<ambientLight intensity={0.5} /\>
|
||||
\<pointLight position={} intensity={1} /\>
|
||||
\<CameraControls ref={cameraControlsRef} /\>
|
||||
|
||||
\<Suspense fallback={null}\>
|
||||
\<group\>
|
||||
{/\* Render all nodes \*/}
|
||||
{nodes.map((node) \=\> (
|
||||
\<Node
|
||||
key={node.id}
|
||||
node={node}
|
||||
onNodeClick={handleNodeClick}
|
||||
/\>
|
||||
))}
|
||||
|
||||
{/\* Render all links \*/}
|
||||
{linkLines.map((line, i) \=\> (
|
||||
\<Line
|
||||
key={i}
|
||||
points={\[line.start, line.end\]}
|
||||
color="\#495057" // gray
|
||||
lineWidth={1}
|
||||
/\>
|
||||
))}
|
||||
\</group\>
|
||||
\</Suspense\>
|
||||
\</Canvas\>
|
||||
);
|
||||
}
|
||||
|
||||
### **Test Specification**
|
||||
|
||||
**1\. Create Test File (tests/magnitude/10-linking.mag.ts)**
|
||||
**1\. Create Test File (tests/magnitude/11-viz.mag.ts)**
|
||||
|
||||
Create a file at /tests/magnitude/10-linking.mag.ts:
|
||||
Create a file at /tests/magnitude/11-viz.mag.ts:
|
||||
|
||||
TypeScript
|
||||
|
||||
import { test } from 'magnitude-test';
|
||||
|
||||
// Helper function to seed the database for this test
|
||||
// Helper function to seed the database
|
||||
async function seedDatabase(agent) {
|
||||
// This would use a custom magnitude.run command or API
|
||||
// to pre-populate the SurrealDB instance with mock nodes.
|
||||
await agent.act('Seed the database with 3 nodes: "Node A", "Node B", "Node C"');
|
||||
// "Node A" is about "dogs and cats"
|
||||
// "Node B" is about "vector databases"
|
||||
// "Node C" is about "ATproto"
|
||||
await agent.act('Seed the database with 5 nodes (A, B, C, D, E) that have embeddings but NO coordinates');
|
||||
}
|
||||
|
||||
test('\[Happy Path\] User can find related links for a draft', async (agent) \=\> {
|
||||
test('\[Happy Path\] User can calculate and view 3D graph', async (agent) \=\> {
|
||||
// Setup: Seed the DB
|
||||
await seedDatabase(agent);
|
||||
|
||||
// Act: Navigate to the editor
|
||||
await agent.act('Navigate to /editor/new');
|
||||
// Act: Go to galaxy page
|
||||
await agent.act('Navigate to /galaxy');
|
||||
|
||||
// Act: Fill out the form with a related idea
|
||||
await agent.act(
|
||||
'Enter "My New Post" into the "Title" input'
|
||||
);
|
||||
await agent.act(
|
||||
'Enter "This idea is about vectors and databases, and how they work." into the "Body" textarea'
|
||||
);
|
||||
// Check: Canvas is empty (no nodes have coords yet)
|
||||
await agent.check('The 3D canvas is visible');
|
||||
await agent.check('The 3D canvas contains 0 node meshes');
|
||||
|
||||
// Act: Click the find related button
|
||||
// (Mock the /api/suggest-links route to return "Node B")
|
||||
await agent.act('Click the "Find Related" button');
|
||||
// Act: Click the calculate button
|
||||
// (Mock the /api/calculate-graph route to return success
|
||||
// and trigger the component re-render)
|
||||
await agent.act('Click the "Calculate My Graph" button');
|
||||
|
||||
// Check: The related node appears in the suggestions
|
||||
await agent.check('A list of suggested links appears');
|
||||
await agent.check('The suggested node "Node B" is visible in the list');
|
||||
await agent.check('The suggested node "Node A" is NOT visible in the list');
|
||||
});
|
||||
|
||||
test('\[Unhappy Path\] User sees empty state when no links found', async (agent) \=\> {
|
||||
// Setup: Seed the DB
|
||||
await seedDatabase(agent);
|
||||
|
||||
// Act: Navigate to the editor
|
||||
await agent.act('Navigate to /editor/new');
|
||||
|
||||
// Act: Fill out the form with an unrelated idea
|
||||
await agent.act(
|
||||
'Enter "Zebras" into the "Title" input'
|
||||
);
|
||||
await agent.act(
|
||||
'Enter "Zebras are striped equines." into the "Body" textarea'
|
||||
);
|
||||
|
||||
// Act: Click the find related button
|
||||
// (Mock the /api/suggest-links route to return an empty array)
|
||||
await agent.act('Click the "Find Related" button');
|
||||
|
||||
// Check: An empty state is shown
|
||||
await agent.check('The text "No related nodes found" is visible');
|
||||
});
|
||||
|
||||
test('\[Happy Path\] User can publish a new node', async (agent) \=\> {
|
||||
// Act: Navigate to the editor
|
||||
await agent.act('Navigate to /editor/new');
|
||||
|
||||
// Act: Fill out the form
|
||||
await agent.act(
|
||||
'Enter "My First Published Node" into the "Title" input'
|
||||
);
|
||||
await agent.act(
|
||||
'Enter "This is the body of my first node." into the "Body" textarea'
|
||||
);
|
||||
// Check: Loading state appears
|
||||
await agent.check('The "Calculate My Graph" button shows a loading spinner');
|
||||
|
||||
// Act: Click Publish
|
||||
// (Mock the /api/nodes route (Commit 06\) to return success)
|
||||
await agent.act('Click the "Publish Node" button');
|
||||
|
||||
// Check: User is redirected to the galaxy
|
||||
await agent.check(
|
||||
'The browser URL is now "http://localhost:3000/galaxy"'
|
||||
);
|
||||
// (After mock API returns and component re-fetches)
|
||||
// Check: The canvas now has nodes
|
||||
await agent.check('The 3D canvas now contains 5 node meshes');
|
||||
});
|
||||
|
||||
test('\[Interaction\] User can click on a node to focus', async (agent) \=\> {
|
||||
// Setup: Seed the DB and pre-calculate the graph
|
||||
await seedDatabase(agent);
|
||||
await agent.act('Navigate to /galaxy');
|
||||
await agent.act('Click the "Calculate My Graph" button');
|
||||
await agent.check('The 3D canvas now contains 5 node meshes');
|
||||
|
||||
// Act: Click on a node
|
||||
// (Magnitude can target R3F meshes by their properties)
|
||||
await agent.act('Click on the 3D node mesh corresponding to "Node A"');
|
||||
|
||||
// Check: Camera moves
|
||||
// (This is hard to check directly, but we can check
|
||||
// for the side-effect: the text label appearing)
|
||||
await agent.check('The camera animates and moves closer to the node');
|
||||
await agent.check('A 3D text label "Node A" is visible');
|
||||
});
|
||||
|
||||
144
docs/voice-mode-implementation-plan.md
Normal file
144
docs/voice-mode-implementation-plan.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# Voice Mode Implementation Plan
|
||||
|
||||
## Phase 1: Clean State Machine
|
||||
|
||||
### Step 1: Rewrite state machine definition
|
||||
- Remove all unnecessary complexity
|
||||
- Clear state hierarchy
|
||||
- Simple event handlers
|
||||
- Proper tags on all states
|
||||
|
||||
### Step 2: Add test buttons to UI
|
||||
- Button: "Skip to Listening" - sends START_LISTENING
|
||||
- Button: "Simulate User Speech" - sends USER_STARTED_SPEAKING
|
||||
- Button: "Simulate Silence" - sends SILENCE_TIMEOUT
|
||||
- Button: "Simulate AI Response" - sends AI_RESPONSE_READY with test data
|
||||
- Button: "Skip Audio" - sends SKIP_AUDIO (already exists)
|
||||
- Display: Current state value and tags
|
||||
|
||||
## Phase 2: Fix Processing Logic
|
||||
|
||||
### Problem Analysis
|
||||
Current issue: The processing effect is too complex and uses refs incorrectly.
|
||||
|
||||
### Solution
|
||||
**Simple rule**: In processing state, check messages array:
|
||||
1. If last message is NOT user with our transcript → submit
|
||||
2. If last message IS user with our transcript AND second-to-last is assistant → play that assistant message
|
||||
3. Otherwise → wait
|
||||
|
||||
**Implementation**:
|
||||
```typescript
|
||||
useEffect(() => {
|
||||
if (!state.hasTag('processing')) return;
|
||||
if (status !== 'ready') return;
|
||||
|
||||
const transcript = state.context.transcript;
|
||||
if (!transcript) return;
|
||||
|
||||
// Check last 2 messages
|
||||
const lastMsg = messages[messages.length - 1];
|
||||
const secondLastMsg = messages[messages.length - 2];
|
||||
|
||||
// Case 1: Need to submit user message
|
||||
if (!lastMsg || lastMsg.role !== 'user' || getText(lastMsg) !== transcript) {
|
||||
submitUserInput();
|
||||
return;
|
||||
}
|
||||
|
||||
// Case 2: User message submitted, check for AI response
|
||||
if (secondLastMsg && secondLastMsg.role === 'assistant') {
|
||||
const aiMsg = secondLastMsg;
|
||||
|
||||
// Only play if we haven't played this exact message in this session
|
||||
if (state.context.lastSpokenMessageId !== aiMsg.id) {
|
||||
const text = getText(aiMsg);
|
||||
send({ type: 'AI_RESPONSE_READY', messageId: aiMsg.id, text });
|
||||
playAudio(text, aiMsg.id);
|
||||
}
|
||||
}
|
||||
// Otherwise, still waiting for AI response
|
||||
}, [messages, state, status]);
|
||||
```
|
||||
|
||||
No refs needed! Just check the messages array directly.
|
||||
|
||||
## Phase 3: Clean Audio Management
|
||||
|
||||
### Step 1: Simplify audio cancellation
|
||||
- Keep shouldCancelAudioRef
|
||||
- Call stopAllAudio() when leaving canSkipAudio states
|
||||
- playAudio() checks cancel flag at each await
|
||||
|
||||
### Step 2: Effect cleanup
|
||||
- Remove submittingTranscriptRef completely
|
||||
- Remove the "reset ref when leaving processing" effect
|
||||
- Rely only on messages array state
|
||||
|
||||
## Phase 4: Testing with Playwright
|
||||
|
||||
### Test Script
|
||||
```typescript
|
||||
test('Voice mode conversation flow', async (agent) => {
|
||||
await agent.open('http://localhost:3000/chat');
|
||||
|
||||
// Login first
|
||||
await agent.act('Log in with Bluesky');
|
||||
|
||||
// Start voice mode
|
||||
await agent.act('Click "Start Voice Conversation"');
|
||||
await agent.check('Button shows "Generating speech..." or "Listening..."');
|
||||
|
||||
// Skip initial greeting if playing
|
||||
const skipVisible = await agent.check('Skip button is visible', { optional: true });
|
||||
if (skipVisible) {
|
||||
await agent.act('Click Skip button');
|
||||
}
|
||||
await agent.check('Button shows "Listening... Start speaking"');
|
||||
|
||||
// Simulate user speech
|
||||
await agent.act('Click "Simulate User Speech" test button');
|
||||
await agent.check('Button shows "Speaking..."');
|
||||
|
||||
await agent.act('Click "Simulate Silence" test button');
|
||||
await agent.check('Button shows "Processing..."');
|
||||
|
||||
// Wait for AI response
|
||||
await agent.wait(5000);
|
||||
await agent.check('AI message appears in chat');
|
||||
await agent.check('Button shows "Generating speech..." or "AI is speaking..."');
|
||||
|
||||
// Skip AI audio
|
||||
await agent.act('Click Skip button');
|
||||
await agent.check('Button shows "Listening... Start speaking"');
|
||||
|
||||
// Second exchange
|
||||
await agent.act('Click "Simulate User Speech" test button');
|
||||
await agent.act('Click "Simulate Silence" test button');
|
||||
|
||||
// Let AI audio play completely this time
|
||||
await agent.wait(10000);
|
||||
await agent.check('Button shows "Listening... Start speaking"');
|
||||
});
|
||||
```
|
||||
|
||||
## Phase 5: Validation
|
||||
|
||||
### Checklist
|
||||
- [ ] State machine is serializable (can be visualized in Stately)
|
||||
- [ ] No refs used in processing logic
|
||||
- [ ] Latest message only plays once per session
|
||||
- [ ] Skip works instantly in both aiGenerating and aiSpeaking
|
||||
- [ ] Re-entering voice mode plays most recent AI message (if not already spoken)
|
||||
- [ ] All test cases from PRD pass
|
||||
- [ ] Playwright test passes
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Add test buttons to UI (for manual testing)
|
||||
2. Rewrite processing effect with simple messages array logic
|
||||
3. Remove submittingTranscriptRef completely
|
||||
4. Test manually with test buttons
|
||||
5. Write Playwright test
|
||||
6. Run and validate Playwright test
|
||||
7. Clean up any remaining issues
|
||||
149
docs/voice-mode-prd.md
Normal file
149
docs/voice-mode-prd.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# Voice Mode PRD
|
||||
|
||||
## User Flows
|
||||
|
||||
### Flow 1: Starting Voice Conversation (No Previous Messages)
|
||||
1. User clicks "Start Voice Conversation" button
|
||||
2. System enters listening mode
|
||||
3. Button shows "Listening... Start speaking"
|
||||
4. Microphone indicator appears
|
||||
|
||||
### Flow 2: Starting Voice Conversation (With Previous AI Message)
|
||||
1. User clicks "Start Voice Conversation" button
|
||||
2. System checks for most recent AI message
|
||||
3. If found and not already spoken in this session:
|
||||
- System generates and plays TTS for that message
|
||||
- Button shows "Generating speech..." then "AI is speaking..."
|
||||
- Skip button appears
|
||||
4. After audio finishes OR user clicks skip:
|
||||
- System enters listening mode
|
||||
|
||||
### Flow 3: User Speaks
|
||||
1. User speaks (while in listening state)
|
||||
2. System detects speech, button shows "Speaking..."
|
||||
3. System receives interim transcripts (updates display)
|
||||
4. System receives finalized phrases (appends to transcript)
|
||||
5. After each finalized phrase, 3-second silence timer starts
|
||||
6. Button shows countdown: "Speaking... (auto-submits in 2.1s)"
|
||||
7. If user continues speaking, timer resets
|
||||
|
||||
### Flow 4: Submit and AI Response
|
||||
1. After 3 seconds of silence, transcript is submitted
|
||||
2. Button shows "Processing..."
|
||||
3. User message appears in chat
|
||||
4. AI streams response (appears in chat)
|
||||
5. When streaming completes:
|
||||
- System generates TTS for AI response
|
||||
- Button shows "Generating speech..."
|
||||
- When TTS ready, plays audio
|
||||
- Button shows "AI is speaking..."
|
||||
- Skip button appears
|
||||
6. After audio finishes OR user clicks skip:
|
||||
- System returns to listening mode
|
||||
|
||||
### Flow 5: Skipping AI Audio
|
||||
1. While AI is generating or speaking (button shows "Generating speech..." or "AI is speaking...")
|
||||
2. Skip button is visible
|
||||
3. User clicks Skip
|
||||
4. Audio stops immediately
|
||||
5. System enters listening mode
|
||||
6. Button shows "Listening... Start speaking"
|
||||
|
||||
### Flow 6: Exiting Voice Mode
|
||||
1. User clicks voice button (at any time)
|
||||
2. System stops all audio
|
||||
3. System closes microphone connection
|
||||
4. Returns to text mode
|
||||
5. Button shows "Start Voice Conversation"
|
||||
|
||||
## Critical Rules
|
||||
|
||||
1. **Latest Message Only**: AI ONLY plays the most recent assistant message. Never re-play old messages.
|
||||
2. **Skip Always Works**: Skip button must IMMEDIATELY stop audio and return to listening.
|
||||
3. **One Message Per Turn**: Each user speech -> one submission -> one AI response -> one audio playback.
|
||||
4. **Clean State**: Every state transition should cancel any incompatible ongoing operations.
|
||||
|
||||
## State Machine
|
||||
|
||||
```
|
||||
text
|
||||
├─ TOGGLE_VOICE_MODE → voice.idle
|
||||
|
||||
voice.idle
|
||||
├─ Check for latest AI message not yet spoken
|
||||
│ ├─ If found → Send AI_RESPONSE_READY → voice.aiGenerating
|
||||
│ └─ If not found → Send START_LISTENING → voice.listening
|
||||
└─ TOGGLE_VOICE_MODE → text
|
||||
|
||||
voice.listening
|
||||
├─ USER_STARTED_SPEAKING → voice.userSpeaking
|
||||
├─ TRANSCRIPT_UPDATE → (update context.input for display)
|
||||
└─ TOGGLE_VOICE_MODE → text
|
||||
|
||||
voice.userSpeaking
|
||||
├─ FINALIZED_PHRASE → voice.timingOut (starts 3s timer)
|
||||
├─ TRANSCRIPT_UPDATE → (update context.input for display)
|
||||
└─ TOGGLE_VOICE_MODE → text
|
||||
|
||||
voice.timingOut
|
||||
├─ FINALIZED_PHRASE → voice.timingOut (restart 3s timer)
|
||||
├─ TRANSCRIPT_UPDATE → (update context.input for display)
|
||||
├─ SILENCE_TIMEOUT → voice.processing
|
||||
└─ TOGGLE_VOICE_MODE → text
|
||||
|
||||
voice.processing
|
||||
├─ (Effect: submit if not submitted, wait for AI response)
|
||||
├─ When AI response ready → Send AI_RESPONSE_READY → voice.aiGenerating
|
||||
└─ TOGGLE_VOICE_MODE → text
|
||||
|
||||
voice.aiGenerating
|
||||
├─ TTS_PLAYING → voice.aiSpeaking
|
||||
├─ SKIP_AUDIO → voice.listening
|
||||
└─ TOGGLE_VOICE_MODE → text
|
||||
|
||||
voice.aiSpeaking
|
||||
├─ TTS_FINISHED → voice.listening
|
||||
├─ SKIP_AUDIO → voice.listening
|
||||
└─ TOGGLE_VOICE_MODE → text
|
||||
```
|
||||
|
||||
## Test Cases
|
||||
|
||||
### Test 1: Basic Conversation
|
||||
1. Click "Start Voice Conversation"
|
||||
2. Skip initial greeting
|
||||
3. Say "Hello"
|
||||
4. Wait for AI response
|
||||
5. Let AI audio play completely
|
||||
6. Say "How are you?"
|
||||
7. Skip AI audio
|
||||
8. Say "Goodbye"
|
||||
|
||||
Expected: 3 exchanges, AI only plays latest message each time
|
||||
|
||||
### Test 2: Multiple Skips
|
||||
1. Start voice mode
|
||||
2. Skip greeting immediately
|
||||
3. Say "Test one"
|
||||
4. Skip AI response immediately
|
||||
5. Say "Test two"
|
||||
6. Skip AI response immediately
|
||||
|
||||
Expected: All skips work instantly, no audio bleeding
|
||||
|
||||
### Test 3: Re-entering Voice Mode
|
||||
1. Start voice mode
|
||||
2. Say "Hello"
|
||||
3. Let AI respond
|
||||
4. Exit voice mode (click button again)
|
||||
5. Re-enter voice mode
|
||||
|
||||
Expected: AI reads the most recent message (its last response)
|
||||
|
||||
### Test 4: Long Speech
|
||||
1. Start voice mode
|
||||
2. Skip greeting
|
||||
3. Say a long sentence with multiple pauses < 3 seconds
|
||||
4. Wait for final 3s timeout
|
||||
|
||||
Expected: All speech is captured in one transcript
|
||||
Reference in New Issue
Block a user